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ABSTRACT 


The  Department  of  Defense’s  proposed  Range  Rule  greatly  increases  the  number 
of  unexploded  ordnance  (UXO)  contaminated  sites  that  the  services  must  decontaminate. 
Existing  models  for  estimating  the  cost  of  UXO  removal  often  require  a  high  level  of 
expertise  and  provide  only  a  point  estimate  for  the  costs;  they  do  not  provide  a 
probability  distribution  of  the  potential  costs.  This  thesis  presents  a  probabilistic  cost 
estimation  model  created  as  an  “add-in”  for  Microsoft  Excel.  A  test  database  consisting 
of  descriptive  and  cost  information  on  the  historic  cleanup  of  nineteen  contaminated  areas 
is  created.  To  demonstrate  the  model,  the  thesis  filters  the  database  to  find  eight  historic 
records  characteristically  similar  to  a  fictitious  cleanup  scenario,  and  uses  information 
from  these  historic  records  to  build  probability  distributions  for  six  cost  elements.  The 
model  applies  Monte  Carlo  simulation  to  these  probability  distributions  to  build  a 
probability  distribution  for  the  total  cleanup  cost.  The  resulting  distribution  shows  that 
for  this  cleanup  scenario  the  most  likely  per  acre  cost  is  $8,400,  but  there  is  a  75%  chance 
that  costs  fall  between  $8,500  and  $26,000.  Results  for  a  scenario  composed  of  three 
cleanups  predicts  a  most  likely  total  cost  of  $1.7  million  with  a  50%  probability  of  costs 
falling  between  $1.7  million  and  $2.2  million. 
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DISCLAIMER 


The  reader  is  cautioned  that  computer  programs  developed  in  this  research  may 
not  have  been  exercised  for  all  cases  of  interest.  While  every  effort  has  been  made, 
within  the  time  available,  to  ensure  that  the  programs  are  free  of  computational  and  logic 
errors,  they  cannot  be  considered  validated.  Any  application  of  these  programs  without 
additional  verification  is  at  the  risk  of  the  user. 
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EXECUTIVE  SUMMARY 


The  Department  of  Defense’s  proposed  Range  Rule  (CFR  32  Part  178)  outlines 
the  military’s  environmental  response  actions  for  non-active  ranges  contaminated  with 
unexploded  ordnance  (UXO).  The  proposed  rule  greatly  increases  the  number  of  UXO 
contaminated  areas  that  the  military  services  must  decontaminate.  The  services  need  to 
plan  the  environmental  remediation  (cleanup)  of  their  contaminated  sites  in  response  to 
the  Range  Rule,  but  most  of  the  military  services  have  limited  experience  in  large-scale 
environmental  restoration.  The  existing  cost  estimating  models  for  ordnance  removal 
are  engineering  cost  models  that  are  based  on  level-of-effort  and  material  costs.  These' 
models  often  require  an  expert’s  judgement  for  the  inputs  and  validation  of  the  outputs. 
Furthermore,  the  current  models  provide  only  a  point  estimate  for  the  costs;  they  do  not 
provide  a  probability  distribution  of  the  potential  costs.  Headquarters  Marine  Corps 
(HQMC)  desires  a  method  to  estimate  cleanup  costs  for  Marine  Corps  sites  affected  by 
the  Range  Rule,  and  an  alternative  to  these  existing  models. 

This  thesis  presents  a  probabilistic  cost  estimation  model  that  allows  for  cost-risk 
analysis.  It  creates  a  test  database  from  nineteen  prior  cleanups  that  are  under  United 
States  Army  Corps  of  Engineers’  management.  The  database  contains  detailed  physical 
descriptions  of  the  contaminated  areas  and  associated  cost  information.  The  database 
segregates  cost  information  by  labor  costs,  material  costs,  and  travel/per  Diem  costs  for 
the  Removal  Action  (RA)  and  the  Environmental  Evaluation/Cost  Analysis  (EE/CA) 
phases  of  cleanup.  This  cost  structure  establishes  six  cost  elements  that  account  for  all 
costs  in  these  final  two  phases  of  a  cleanup. 


XIX 


The  model’s  design,  as  an  “add-in”  for  Microsoft  Excel,  makes  it  accessible  to 
most  military  users  and  provides  a  user-friendly  means  to  filter  records  and  find  a  pool  of 
historic  sites  that  are  characteristically  similar  to  a  proposed  cleanup.  Prior  research  and 
analysis  of  the  data  shows  that  environmental  characteristics  of  the  contaminated  area, 
characteristics  of  the  UXO,  proposed  future  use,  and  cleanup  methodology,  determine 
cleanup  costs.  Once  the  user  finds  an  appropriate  pool  of  data  using  these  predictive 
factors,  the  model  assists  the  user  in  analyzing  the  historic  cost  information  and  building 
probability  distributions  for  the  six  cost  elements.  The  model  uses  a  weighted 
distribution  for  each  cost  element  that  is  a  linear  combination  of  the  empirical  data  and  a 
user  supplied  triangular  distribution.  Monte  Carlo  simulation  sums  the  weighted 
probability  distributions  of  the  six  cost  elements  in  order  to  build  a  total  cost  probability 
distribution  for  the  RA  phase  and  the  EE/CA  phase  of  a  site  restoration. 

A  fictitious  cleanup  scenario  demonstrates  the  model’s  functionality.  The 
resulting  total  cost  probability  distribution  reveals  that  for  this  cleanup  scenario  the  most 
likely  cost  is  $8,400  per/acre,  but  the  distribution  also  shows  that  there  is  roughly  a  75% 
chance  that  costs  actually  fall  between  $8,500  and  $26,000.  The  thesis  further 
demonstrates  how  Monte  Carlo  Simulation  can  aggregate  the  total  cost  estimates  for 
several  related  areas  to  produce  a  single  multi-area  estimate.  Results  for  a  scenario 
composed  of  three  contaminated  areas  predict  a  most  likely  cost  of  $1 .7  million  with  a 
50%  probability  of  costs  falling  between  $1.7million  and  $2.2million. 

This  work  recommends  that  future  requirements  standardize  and  automate  the 
reporting  of  cost  information  by  contractors.  The  limited  amount  of  available  cost  data 


XX 


reduces  confidence  in  the  estimates  that  this  model  produces.  More  cost  data  can  provide 
the  user  with  a  larger  pool  of  similar  cleanups  and  can  reduce  the  assumptions  that  the 
user  must  make  in  performing  an  estimate.  This  thesis  also  recommends  that  the 
Department  of  Defense  consider  implementing  cost-risk  analysis  in  other  models  that  it 
uses  for  estimating  UXO  cleanup  costs. 

Although  HQMC  is  the  impetus  for  the  development  of  this  model,  the  cost-risk 
model  is  suitable  for  all  Department  of  Defense  users  interested  in  producing  cost 
estimates  for  UXO  cleanup.  The  thesis  provides  an  easy-to-use  model  that  allows  the 
Marine  Corps  and  other  users  to  budget  for  the  cleanup  of  UXO.  Estimating  the  costs  of 
environmental  restoration  of  ordnance  contaminated  sites  is  a  difficult  process  containing 
a  substantial  amount  of  unknowns.  The  cost-risk  model  provides  the  means  to  manage 
this  uncertainty  and  risk. 
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I.  INTRODUCTION 


The  Department  of  Defense’s  proposed  Range  Rule  (CFR  32  Part  178)  outlines 
the  military’s  environmental  response  actions  for  non-active  ranges  contaminated  with 
unexploded  ordnance  (UXO).  By  publishing  an  acceptable  rule  that  governs  the  handling 
of  military  munitions  on  closed,  transferred  and  transferring  ranges,  the  DOD  hopes  to 
avoid  having  to  treat  UXO  as  solid  hazardous  waste.  The  proposed  rule,  however, 
greatly  increases  the  required  number  of  UXO  contaminated  areas  that  the  military 
services  must  clean.  The  services  need  to  plan  the  environmental  remediation  (cleanup) 
of  their  contaminated  sites  in  response  to  the  draft  rale,  but  with  the  exception  of  the 
Army  the  military  services  have  limited  experience  in  large-scale  environmental 
restoration.  The  Army  Corps  of  Engineers,  through  its  involvement  in  the  Defense 
Environmental  Restoration  Program  -  Formerly  Used  Defense  Sites  (DERP-FUDS)  and 
Base  Realignment  and  Closure  (BRAC),  has  extensive  experience  in  remediation  of  UXO 
sites.  The  existing  cost  estimating  models  for  ordnance  removal  projects  are  engineering 
cost  models  that  are  based  on  level-of-effort  and  material  costs.  These  models  often 
require  an  expert’s  judgement  for  the  inputs  and  validation  of  the  outputs.  Further,  the 
current  models  provide  only  a  point  estimate  for  the  costs;  they  do  not  provide  a 
probability  distribution  of  the  possible  costs. 

This  thesis  presents  a  probabilistic  cost  estimation  model  that  provides  the 
military  services  the  ability  to  perform  cost-risk  analysis.  The  user  can  develop  a  cost 
estimate  using  the  cost  data  from  historic  sites  that  have  characteristics  similar  to  the 
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proposed  cleanup  site.  The  model  produces  a  probability  distribution  for  the  cost  of  a 
single  cleanup  or  several  combined  cleanups.  These  distributions  allow  a  user  to  budget 
for  the  cleanup  of  UXO  on  a  specific  site  and  to  make  decisions  on  how  to  approach  the 
remediation  of  multiple  sites. 

Chapter  n  provides  an  overview  of  the  UXO  problem  and  current  methods  of 
estimating  UXO  cleanup  costs.  Included  in  Chapter  n  is  related  research  in  probabilistic 
cost  estimating  and  general  cost  estimating  of  environmental  projects.  Chapter  HI 
outlines  cost  factors  in  the  UXO  remediation  process.  Chapter  IV  presents  background 
information  necessary  for  understanding  the  probabilistic  cost  model,  and  this  chapter 
gives  a  detailed  description  of  the  data  used  in  the  model  and  discusses  the  general 
method  of  Monte  Carlo  Sampling  used  to  produce  total  cost  probability  distributions.  In 
Chapter  V  a  “walkthrough”  of  the  model  is  performed  using  a  cleanup  scenario  to 
illustrate  the  model's  functionality.  Discussion  of  the  simulation  output  and  its 
interpretation  is  given  in  Chapter  VI,  and  finally  Chapter  Vn  delivers  recommendations 
and  conclusions. 
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II.  BACKGROUND 


A.  SCOPE  OF  THE  UXO  PROBLEM 

UXO  is  a  massive  challenge;  worldwide,  nearly  500  people  a  week  are  killed  or 
injured  from  a  staggering  100  million  landmines  left  over  from  prior  conflicts  [United 
States  Department  of  Defense,  Office  of  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology  (USD  (A&T))  1998].  Although  in  the  United  States  death  or  injury  is 
extremely  rare,  UXO  has  other  substantial  detriments.  Estimates  show  that  some  level  of 
UXO  contaminates  nearly  15  million  acres  in  the  United  States.  This  contamination 
prevents  civilian  land  use,  threatens  public  safety,  and  causes  environmental  concern. 
Since  many  suspect  sites  have  not  been  surveyed,  the  extent  of  this  problem  is  uncertain. 
However,  estimates  show  that  if  only  five  percent  of  these  sites  require  remediation, 
cleanup  costs  could  exceed  15  billion  dollars  [USD  (A&T)  1998]. 

To  address  the  UXO  problems,  the  Environmental  Protection  Agency  (EPA) 
published  the  Military  Munitions  Rule  in  February  of  1997.  The  EPA,  however, 
postponed  a  decision  on  whether  to  regulate  UXO  on  closed,  transferred,  and  transferring 
(CTT)  ranges  while  the  Defense  Department  develops  its  own  Range  Rule  for  these  CTT 
ranges.  If  the  DOD  does  not  promulgate  the  Range  Rule  or  if  the  Range  Rule  does  not 
adequately  protect  the  public  and  the  environment,  the  EPA  plans  to  force  cleanup  on 
these  CTT  ranges  under  United  States  environmental  law  [United  States  Department  of 
Defense,  Office  of  the  Secretary  1997,  sec.  n.l.].  By  drafting  the  Range  Rule,  the  DOD 
hopes  to  uniquely  define  UXO  from  other  hazardous  wastes,  and  establish  methods  to 
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handle  UXO  that  is  acceptable  to  the  military,  EPA,  and  other  stakeholders.  The 
different  services  understand  that  unless  they  voluntarily  demonstrate  responsible 
stewardship  of  these  lands,  as  addressed  in  the  Range  Rule,  restrictions  could  be  imposed 
that  would  severely  limit  training  and  create  insurmountable  costs. 

Due  to  the  implications  of  the  proposed  Range  Rule,  the  services  are  showing 
concern  over  the  many  issues  surrounding  UXO  management  and  removal  including  the 
analysis  of  cleanup  costs.  One  reason  that  the  estimation  of  cleanup  costs  proves  to  be 
difficult  is  that  there  is  tremendous  variance  found  in  the  costs  from  one  site  to  another. 
For  example,  the  budget  for  the  Navy’s  Kaho’olawe  range  in  Hawaii  is  $400  million  or 
approximately  $14,000  per  acre;  the  Army’s  Jefferson  Proving  Ground  estimates  are  as 
high  as  $1.4  million  dollars  per  acre;  and  parcels  of  Adak,  Alaska  have  estimates  up  to  a 
staggering  $10  million  per  acre.  [Rohrer  1997] 

B.  CURRENT  STATUS  OF  UXO  COST  ESTIMATION 

United  States  Army  Engineering  and  Support  Center  Huntsville,  Alabama 
USAESCH)  manages  and  has  managed  the  majority  of  UXO  remediation  projects  in  the 
United  States.  Throughout  the  1970’s  and  early  1980’s  the  DERP-FUDS  evolved.  This 
program’s  goal  is  to  remove  any  risk  to  human  health,  safety,  and  the  environment  caused 
by  former  DOD  activities.  The  DOD  delegates  the  execution  of  this  program  to  the 
United  States  Army  Corps  of  Engineers  (US  ACE).  As  the  chief  executor  of 
environmental  activities  at  PTJDS,  the  USACE  is  involved  in  a  varying  capacity  of 
cleanup  on  approximately  1,500  FUDS.  To  create  a  focal  point  for  all  UXO  activities  the 
USACE  created  a  Mandatory  Center  of  Excellence  (MCX)  at  Huntsville,  Alabama  in 
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1990.  Along  with  the  DERP-FUDS,  the  MCX  is  also  responsible  for  managing  UXO 
activities  on  BRAC  sites,  and  Installation  Restoration  Program  sites.  In  total  the  MCX 
has  over  2,000  inventoried  sites  and  usually  administers  from  60  to  80  restoration 
projects  at  any  given  time  [Foley  et  al.  1996]. 

Huntsville  Center  produces  cost  estimates  for  UXO  projects  using  either  the 
Remedial  Action  Cost  and  Engineering  Requirements  System  (RACER)  [BTG 
Incorporated  (Delta  Research  Division)  1998],  or  a  proprietary  spreadsheet  model 
developed  by  Huntsville  Center’s  cost  engineers.  Both  RACER  and  the  spreadsheet 
model  base  calculations  on  the  number  of  teams  that  a  project  requires  for  UXO  removal. 
The  two  models  then  determine  total  labor  hours,  equipment  use,  and  material 
requirements  based  on  these  hypothetical  teams.  Finally,  these  two  models  use  labor-rate 
costs  and  material  unit  costs  to  arrive  at  a  single  point  estimate.  Unfortunately,  these 
models  do  not  provide  any  information  about  cost  uncertainty  and  ^so  require  expert 
knowledge  of  UXO  remediation  in  order  to  verify  inputs  and  results.  The  USACE  uses 
experienced  cost  engineers  to  produce  cost  estimates.  For  other  agencies  these  models 
would  be  difficult  to  use  without  background  experience  in  UXO  projects. 

C.  RELATED  RESEARCH 

Throughout  the  DOD  cost  analysts  frequently  use  cost-risk  analysis  models,  and 
the  research  in  this  field  is  abundant.  Anderson  and  Cherwonik  [1997]  present  an 
overview  of  cost-risk  analysis  in  military  acquisition  programs.  Their  primer  describes 
the  current  DOD  shift  away  from  point  estimates  to  probabilistic  estimates,  and  justifies 
the  need  for  this  shift.  The  paper  focuses  on  providing  the  acquisition  Program  Manager 
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and  other  integral  DOD  acquisition  personnel  with  an  appreciation  of  the  benefits  of  cost- 
risk  analysis. 

More  specifically  related  to  this  thesis,  the  DOD  uses  cost  analysis  directly  in 
environmental  programs.  The  MITRE  Corporation,  a  subcontractor  in  the  Army’s 
chemical  weapons  demilitarization  program,  presents  a  methodology  that  compares  two 
different  processes  used  for  demilitarizing  chemical  weapons  [Hughitt,  Wusterbath,  and 
Zoepfl  1994].  The  MITRE  Corporation  report  uses  Latin  hypercube  sampling  of  work 
breakdown  structure  cost-elements  to  create  total  cost  probability  distributions  for  the 
demilitarization  process.  The  report  then  compares  the  life  cycle  costs  of  the  current 
baseline  process  with  the  proposed  cryofracture  process.  Since  the  two  processes  are  in 
different  stages  of  developmental  maturity,  MITRE  develops  input  probability 
distributions  with  a  combination  of  theoretical  and  empirical  data.  The  results  of  the 
analysis  provide  a  method  to  compare  the  economic  merits  of  the  two  plant  designs  and 
their  associated  demilitarization  processes. 

In  other  research.  Air  Force  Materiel  Command  contracted  Aeronautical  Systems 
Center  to  develop  a  funding  estimator  tool  for  pollution  prevention  in  weapon  system 
production  [Aeronautical  Systems  Center  1994].  The  EPA  requirements  to  reduce  the 
use  of  ozone  depleting  chemicals  forces  program  managers  to  develop  strategies  and 
funding  plans  to  meet  the  EPA  goals.  The  Air  Force  Materiel  Command  model  provides 
a  means  to  produce  rough  order  of  magnitude  funding  estimates  to  support  Program 
Objective  Memorandum  inputs.  The  model  identifies  a  weapon  system’s  logistic  and 
manufacturing  processes  and  the  hazardous  material  associated  with  these  processes. 
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Next,  the  model  assigns  each  hazardous  material  involved  in  the  production  process  an 
importance  factor,  difficulty  factor,  and  substitute  factor.  Finally,  the  model  uses 
regression  analysis  to  provide  pollution  prevention  funding  estimates. 

Hersey  and  Pederson’s  unpuplished  report  is  one  of  the  earliest  works  specifically 
addressing  UXO  removal  costs  [1981].  Their  work  outlines  the  factors  that  influence 
cleanup  costs:  characteristics  of  the  contaminated  area,  characteristics  of  the  ordnance, 
cleanup  requirements,  and  the  cleanup  methodology.  They  present  an  engineering 
“bottom-up”  model  that  provides  a  means  to  compare  the  cost  of  manual  removal  with 
removal  assisted  by  a  Sub-Surface  Clearance  Vehicle.  The  model  compares  the  costs  of 
the  two  removal  methods  under  varying  environmental  conditions. 

Delta  Research,  a  division  of  BTG  Incorporated,  has  done  the  greatest  amount  of 
research  in  estimating  costs  of  UXO  removal.  Delta  Research’s  team  of  environmental 
engineers,  cost  analysts,  contractors,  and  economists  developed  the  RACER  System  to 
provide  the  DOD  and  the  private  sector  with  a  system  to  assist  in  estimating  costs  of 
hazardous  waste  cleanup.  RACER  consists  of  over  65  separate  engineering  cost  and 
project  management  models,  with  a  few  dedicated  solely  to  ordnance  and  explosive  waste 
remediation.  Delta  Research  consolidates  the  UXO  models  into  a  separate  Microsoft 
Excel  [  Microsoft  Corporation  1997]  based  package  titled  Ordnance  Remediation  Cost 
Analyzer  (ORCA).  These  models  are  called  RACER/ORCA  in  the  remainder  of  this 
thesis.  These  models  calculate  quantities  of  equipment,  labor,  and  materials  needed  for  a 
job  based  on  the  underlying  engineering  processes  [BTG  Incorporated  (Delta  Research 
Division)  1999]. 
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III.  ORDNANCE  REMOVAL  PROCESSES  AND  COST  DRIVERS 


A.  PHASES  OF  RESTORATION 

UXO  remediation  usually  follows  three  phases:  “inventory,”  “study,”  and 
“removal.”  The  total  cost  of  site  restoration  from  initial  project  identification  through 
ordnance  removal  is  known  as  the  “cost  to  complete”  (CTC).  Estimating  the  CTC 
requires  understanding  the  costs  in  each  phase  of  a  cleanup. 

1.  Inventory  Phase 

For  a  DERP-FUDS  project,  the  inventory  phase  begins  with  the  development  of 
an  Initial  Project  Report  (INPR).  The  INPR  determines  whether  a  threat  exists,  whether 
the  site  is  the  responsibility  of  the  DOD,  and  whether  the  site  is  eligible  for  cleanup  under 
DERP-FUDS.  The  cost  for  an  INPR  is  usually  relatively  small  (less  than  $25,000) 
[Fanning  1999]. 

2.  Study  Phase 

The  study  phase  usually  consists  of  two  tasks:  the  Archive  Search  Report  (ASR) 
and  the  Engineering  Evaluation/Cost  Analysis  (EE/CA).  The  ASR  is  a  detailed 
investigation  of  historic  information  related  to  the  suspected  contaminated  areas.  The 
agency  conducting  an  ASR  reviews  any  existing  paperwork,  conducts  interviews,  reviews 
maps,  and  visually  examines  the  suspected  and  surrounding  areas.  Producing  an  ASR 
with  reliable  conclusions  can  be  quite  difficult  because  most  military  ranges  have  few  if 
any  records  on  their  use.  Range  Boundaries  shift  over  the  years,  ranges  overlay  onto 
other  ranges,  and  ranges  convert  to  alternate  use.  To  compound  this  problem,  ranges  are 
often  not  segregated  by  use  and  are  open  to  all  types  of  ordnance  and  ordnance  delivery 
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[Hersey  and  Pederson  1981].  The  final  ASR  product  includes  maps  of  the  suspected 
area,  a  detailed  description  of  the  physical  composition  of  these  areas,  and  a 
determination  of  the  types  and  densities  of  the  ordnance  that  may  be  encountered.  The 
Saint  Louis  district  or  Rock  Island  district  of  the  Corps  of  Engineers  normally  completes 
ASRs.  Regardless  of  the  project’s  size,  an  ASR  usually  has  a  fixed  price  of  about 
$40,000  [Fanning  1999]. 

When  the  ASR  identifies  areas  of  contamination,  then  the  study  phase  conducts 
an  EE/CA.  The  EE/CA  is  a  comparative  study  of  removal  options  performed  by  the 
contractor.  A  large  part  of  the  EE/CA’ s  cost  is  the  initial  sampling  of  the  site.  The 
sampling  results  subdivide  the  contaminated  area  into  several  sectors  based  on  ordnance 
densities  (number  of  ordnance  per  acre).  Sampling  usually  involves  choosing  randomly 
located  grids  within  a  sector  and  surveying  them  with  a  magnetometer.  All  anomalies 
are  then  excavated  to  determine  density.  The  contractor  uses  the  sampling  data  to 
perform  analysis  on  removal  action  alternatives  and  to  develop  plans  for  the  Removal 
Action  phase.  EE/CA  costs  can  be  substantial.  Current  algorithms  recommend  sampling 
a  minimum  of  two  percent  of  a  sector  in  order  to  determine  ordnance  density  with  a 
reasonable  degree  of  confidence  [Barrett  1999]. 

3.  Removal  Action  Phase 

Huntsville  Center  estimates  that  Removal  Action  (RA),  an  extremely  labor- 
intensive  operation,  constitutes  roughly  80%  of  the  CTC  [Young  1999].  Most  RAs  are 
“mag-and-flag”  operations.  Highly  trained  ordnance  technicians  divide  the  entire  area  of 
contamination  into  grids,  and  then  subdivide  the  grids  into  approximately  20-foot  wide 
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lanes.  The  technicians  sweep  the  lanes  with  handheld  magnetometers  and  place  flags 
wherever  an  anomaly  in  the  earth’s  magnetic  field  is  registered.  Finally,  the  technicians 
use  manual  methods  to  dig  up  all  anomalies.  A  newer  method,  called  geophysical 
surveying,  consists  of  sweeping  the  entire  area  with  systems  that  automatically  record 
anomaly  locations  and  signatures.  The  data  is  then  analyzed  to  filter  out  false  readings 
from  actual  UXO.  Unlike  mag-and-flag,  geophysical  operations  only  remove  those 
anomalies  determined  to  be  UXO.  Geophysical  methods  can  greatly  reduce  the  cost  of 
the  RA  by  reducing  the  amount  of  “digs”  to  perform;  however,  geophysical  surveying 
adds  to  costs  in  the  EE/CA  stage  because  of  the  additional  resources  and  expertise 
required  in  the  investigation  phase. 

All  of  the  services  have  Explosive  Ordnance  Disposal  (EOD)  units;  however, 
large-area  environmental  restoration  differs  in  many  ways  from  the  typical  missions 
performed  by  military  EOD  units.  Due  to  this  and  the  operational  commitments  on  EOD 
units,  civilian  contractors  perform  nearly  all  Removal  Actions  on  environmental 
restoration  projects. 

B.  FACTORS  AFFECTING  COST 

The  factors  that  influence  cost  are  well  known  throughout  the  UXO  community, 

but  the  interrelationships  of  these  factors  are  not  fully  understood.  An  economic  model 
developed  by  members  of  the  Naval  Explosive  Ordnance  Disposal  Technical  Center  in 
the  early  1980s,  shows  the  cost  of  range  cleanup  to  be  b^sed  on  four  factors:  proposed 
future  use,  environmental  characteristics  of  the  site,  expended  ordnance  characteristics, 
and  method  of  removal  [Hersey  and  Pederson  1981]. 
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1. 


Proposed  Future  Use  and  Depth  of  Clearance 


Proposed  future  use  is  one  of  the  primary  factors  in  the  cleanup  cost  because  it 
dictates  the  depth  of  removal.  Table  1  shows  clearance  depths  prescribed  by  the  DOD  for 


varying  categories  of  future  land  use. 


Projected  Future  Use 

Depth  of  Clearance] 

Unrestricted  _ . 

■■■nifJSflHHI 

Commercial  /  Residential  /  Utility  /Subsurface  Recreational 

Construction  Activity 

Public  Access 

4  feet 

Farming  /  Agriculture  /  Surface 

Recreation  /  Vehicle  Parking  /  Surface 

Supply  Storage 

Limited  Public  Access 

1  foot 

Livestock  Grazing  /  Wildlife  Preserve 

Not  Yet  Determined 

Surface 

Table  1:  Projected  Future  Use  and  Required  Depth,  of  Cleanup.  The  depths  listed 
in  this  table  are  default  depths  for  ordnance  removal.  Actual  depths  may  differ  in 
accordance  with  contractor,  DOD,  and  stakeholder  negotiations.  From  [United 
States  Department  of  Defense,  Office  of  the  Secretary  1995]. 

As  the  depth  of  clearance  increases,  the  technology  needed  to  locate  UXO  changes  and 

the  methods  to  excavate  it  become  more  complex.  Although  not  necessarily  indicative  of 

all  projects,  the  following  table  of  projected  costs  to  cleanup  Naval  Station  Adak,  Alaska 

shows  exponential  cost  growth  associated  with  increases  in  clearance  depth. 
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Depth  Of  Clearance 

Cost  in  Dollars/Acre 

Surface 

$2,260 

IFt 

$6,371 

4  Ft 

$24,715 

10  Ft 

$2,086,400 

Table  2:  Estimated  Costs  for  Varying  Depth  of  Clearance  on  Adak,  Alaska.  As 
depth  of  clearance  increases,  estimated  costs  per  foot  increase  exponentially.  Note 
that  a  four-foot  clearance  is  approximately  ten  times  the  cost  of  a  surface  clearance 
while  the  ten-foot  clearance  is  nearly  one  thousand  times  more  expensive  [Rohrer 
1997]. 

2.  Environmental  Characteristics 

The  environmental  characteristics  of  a  contaminated  area  including  topography, 
vegetation,  and  soil  type  also  influence  cost.  The  topography  of  the  site  may  determine 
what  methods  of  surveying  and  removal  to  use.  Very  hilly  or  mountainous  terrain  can 
limit  both  detection  and  removal  to  man-portable  means  only.  Similarly,  extremes  such 
as  frozen  tundra  or  swamps  limit  certain  technologies  and  removal  methods.  Vegetation 
of  the  range  can  also  determine  cost;  often  it  must  be  removed  prior  to  surveying  and 
ordnance  removal.  For  example,  the  dense  jungle  hampers  the  clearance  of  Panamanian 
ranges  [Panama  Canal  Treaty  Implementation  Plan  Agency  1997]  while  even  the  simple 
shrubs  of  the  northern  California  coast  make  the  clearance  of  the  former  Fort  Ord  costly 
and  difficult  [Meuser,  and  Szasz  1997].  Finally,  soil  type  affects  the  maximum  depth  that 
ordnance  penetrates,  the  ability  of  detectors  to  locate  UXO,  and  the  level  of  difficulty  of 
removing  ordnance.  For  example,  ordnance  penetrates  far  deeper  into  soft  sand  or  silt 
then  into  hard  clay  or  rock.  Similarly,  it  is  far  more  difficult  to  locate  ordnance  in  clay 
and  rocky  soils  because  the  heavy  mineral  content  often  produces  false  anomalies  during 
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magnetic  surveys.  Finally,  it  is  physically  more  difficult  to  excavate  ordnance  in  clay  and 
rocky  soils  than  in  softer  soils. 

3.  Ordnance  Characteristics 

Density  (number  of  UXO  per  acre)  and  the  types  of  ordnance  present  are  also 
factors  that  influence  cleanup  costs.  The  density  of  the  ordnance  corresponds  directly  to 
the  amount  of  excavation  needed  and  the  overall  amount  of  work  to  be  performed.  More 
items  in  the  ground  means  more  holes  need  to  be  dug,  and  more  items  unearthed  means 
more  items  to  treat  and  dispose  of.  The  poor  status  of  current  detection  technology 
compounds  the  costs  associated  with  higher  density  because  the  false  alarm  rate  is  often 
as  high  as  six  false  alarms  for  every  actual  piece  of  ordnance  [USD  (A&T)  1998].  The 
type  of  ordnance  and  its  delivery  method  also  affects  how  deep  the  ordnance  penetrates 
the  earth.  Finally,  removing  different  t)^s  of  ordnance  clearly  involves  different  means 
of  removal.  DOD  ranges  contain  anything  from  small  grenades  a  few  inches  long  to 
2,000  pound  bombs  several  feet  in  length.  In  particular,  chemical  weapons  require 
specialized  removal  methods.  Figure  1  depicts  the  myriad  of  ordnance  that  ordnance 
technicians  encounter  in  UXO  removal. 
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Figure  1:  Common  Types  of  UXO  and  UXO  Categories.  Different  types  of  UXO 
impose  a  varying  degree  of  difficulty  for  their  removal.  UXO  type  is  thus  an 
indicator  of  the  cost  of  UXO  removal.  From  [USD(A&T),1998]. 


4.  Removal  Methods 

The  cost  factor  generating  the  most  interest  and  debate  among  experts  is  removal 
methods  and  removal  technologies.  As  previously  discussed,  current  methods  for  UXO 
removal  are  labor  intensive  manual  operations.  A  Defense  Science  Board  report  [United 
States  Department  of  Defense,  Office  of  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology  (USD  (A&T))  1998]  concludes  that  approximately  $125  million  is  spent 
yearly  by  the  DOD  for  UXO  cleanup  and  that  $70-$80  million  of  this  cost  is  for  labor. 
The  report  recommends  a  coordinated  research  and  development  (R&D)  program  and  a 
doubling  of  the  current  amount  spent  on  R&D.  The  Board  believes  that  labor  costs  can 
be  drastically  reduced  by  the  implementation  of  automating  technologies.  As  the  report 
states; 
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Despite  limitations  of  the  current  R&D  program,  there  is  enough 
scientific  understanding  and  experimental  data  to  convince  the  Task  Force 
that  an  aggressive  and  well  managed  program  could  demonstrate  dramatic 
improvements  in  cost  effectiveness  within  the  next  few  years.  [USD 
(A&T)  1998] 

The  technologies  being  developed  span  the  complete  array  of  remediation 
requirements.  Although  the  focus  is  on  improving  sensor  capabilities  used  in  detection, 
other  technologies  involve  computer  systems  for  automatic  and  digital  mapping  of  a 
contaminated  area,  robots  and  advanced  excavators  for  mechanized  removal,  and 
artificial  intelligence  knowledge-bases  for  discriminating  UXO  anomalies.  Some  of 
these  emerging  technologies  are  slowly  being  adopted;  however,  substantiated  cost 
relationships  based  on  these  technologies  do  not  exist. 
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rv.  MODEL  DEVELOPMENT 


A.  EXISTING  ORDNANCE  DATABASES  AND  MANAGEMENT  SYSTEMS 

Huntsville  Center  contracted  Computer  Systems  Technologies  to  develop  the 
Ordnance  and  Explosive  Database  (OED)  [Computer  Systems  Technology  1996],  but  the 
contract  was  not  fully  completed.  The  database  was  intended  to  record  historic 
information  on  all  ordnance  removal  projects,  provide  a  means  to  analyze  information  for 
use  in  future  cost  estimates,  and  provide  a  cornerstone  for  an  overall  project  management 
system.  Currently  the  database  is  sparsely  populated  and  contains  no  cost  information. 
The  OED  requires  further  development  to  become  a  mature  system  capable  of  fulfilling 
the  needs  of  the  USAESCH. 

Besides  the  OED,  two  other  DOD  projects  propose  to  develop  an  ordnance 
management  system  that  contain  information  on  all  known  and  potential  sites 
contaminated  with  UXO.  A  partnership  between  the  Defense  Environmental  Security 
Corporate  Information  Management  Program  Office  and  the  Department  of  Defense 
Explosive  Safety  Board  is  sponsoring  the  development  of  the  Defense  Explosive 
Management  Suite  which  contains  a  module  for  ordnance  management  called 
Unexploded  Ordnance  Management  [Hoehl  1997].  The  other  proposed  system  that 
incorporates  overall  program  management  is  the  Range  Information  Management  System 
(RIMS)  [Martino  et  al.  1998].  Although  all  three  systems  are  in  early  stages  of 
development,  these  projects  suggest  that  some  future  system  will  exist  that  contains 
detailed  ordnance  cost  information.  This  thesis  designs  a  cost-risk  model  to  interface 
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with  the  OED,  but  the  theory  can  be  used  to  design  a  cost  analysis  module  that  can  be 
incorporated  into  any  future  management  system. 

B.  DATA  DESCRIPTION 

Since  the  OED  is  not  populated  with  cost  data,  the  author  gathered  cost 
information  from  USAESCH  reports  and  created  a  test  database  that  attempts  to  parallel 
the  OED  design.  Final  Removal  Action  Reports,  EE/CA  Reports,  and  project  manager 
flies  are  the  sources  for  cost  information.  Currently  collecting  cost  information  from 
these  sources  is  a  slow  manual  process,  and  this  limits  the  amount  of  cost  records 
available  for  this  thesis.  Only  recently  has  there  been  a  requirement  for  the  contractors 
to  deliver  information  in  a  digital  form.  Most  information  exists  only  in  paper  reports 
stored  in  Huntsville  Center’s  warehouse. 

In  relation  to  UXO  program  management,  Huntsville  Center  defines  a  “site”  as 
DOD  property  that  contains  UXO.  A  military  base,  proving  ground,  or  test  range  is  a 
site.  A  “project”  is  a  subset  of  a  site  where  some  phase  of  remediation  has  occurred  or  is 
occurring.  A  site  can  have  multiple  projects  each  with  a  separate  project  manager.  An 
“Ordnance  Operable  Unit  (OOU)”  is  a  subset  of  a  project.  An  OOU  is  an  ordnance- 
contaminated  area  that  has  homogenous  physical  characteristics  and  properties.  Usually 
the  removal  action  is  similar  throughout  the  OOU.  Since  a  project  can  contain  several 
very  diverse  OOUs,  cost  data  must  be  captured  at  the  OOU  level  in  order  to  compare  two 
cleanups.  This  thesis  collects  the  costs  for  Removal  Action  of  19  OOUs.  These  19 
OOUs  comprise  12  different  projects  from  1 1  separate  sites.  Table  3  shows  the 
breakdown  of  the  data  set. 
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Site  Location 

Number  of  OOUs  per  Project 

Plattsburgh  AFB,  NY 

2 

Miramar  NAS,  CA 

2 

Camp  Croft,  SC  project  1 

5 

Camp  Croft,  SC  project  2 

1 

Fort  Sheridan,  IL 

1 

Camp  Maxey,  TX 

1 

Fort  Devan,  MA 

1 

Savanna  Army  Dep,  IL 

1 

Fort  Wingate,  NM 

1 

Erie  Ord  Depot,  OH 

1 

Duck  Target  Facility,  NC 

1 

Fort  Ord,  CA 

1 

Table  3:  Breakdown  of  Data  by  Site.  The  thesis  uses  cost  data  for  19  OOUs  to 
demonstrate  the  cost-risk  model.  The  site  location  and  the  number  of  OOUs  from 
each  project  are  shown.  Camp  Croft  is  the  only  site  used  with  data  available  for 
more  than  one  project.  The  data  for  each  site  reflects  information  that  was 
available  during  data  collection,  and  may  not  encompass  all  projects  for  the  site. 

The  larger  sites  such  as  Fort  Ord  may  contain  more  ongoing  or  completed  projects, 
but  the  reports  were  not  found. 

The  OED’s  design  uses  a  72-line  work  breakdown  structure  (WBS)  for  reporting  costs  in 
all  phases  of  remediation.  Although  this  type  of  detail  is  desired,  there  is  no  requirement 
for  the  contractors  to  report  costs  with  this  level  of  fidelity.  Until  requirements  exist  that 
force  contractors  to  report  costs  in  this  format,  the  OED’s  cost  input  structure  may  be  an 
unrealistic  design.  Cost  data,  however,  is  routinely  reported  in  “labor,”  “material,”  and 
“travel/per  Diem”  costs.  This  thesis,  therefore,  collects  cost  data  for  each  OOU  using  this 
breakdown.  The  thesis  also  assumes  that  these  costs  can  be  converted  to  costs  per  acre 
because  they  are  generally  variable  costs  dependent  on  the  length  of  time  to  complete  a 
project.  Travel  costs,  however,  are  fixed  costs  that  depend  little  on  the  size  of  the 
cleanup.  The  travel  costs  could  not  be  separated  from  per  Diem  costs,  but  in  general 
travel  is  a  small  fraction  of  the  overall  ‘Travel/per  Diem”  costs.  This  breakdown 
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provides  enough  fidelity  for  estimates,  but  the  cost-risk  model  can  be  adapted  to  handle 
higher  fidelity  WBSs. 

C.  COST-RISK  ANALYSIS 

The  majority  of  the  remainder  of  this  chapter  provides  a  general  background  of 
cost-risk  analysis.  Readers  well  versed  in  cost-risk  analysis  may  skip  to  Subsection 
rV.C.4.  Weighted  Estimate  of  a  Distribution  Function.  Much  of  the  background 
discussion  on  cost-risk  analysis  in  this  chapter  summarizes  information  contained  in  “The 
Probabilistic  Approach  to  Cost  Analysis”  [Book  1999]  and  “Cost-risk  and  Cost 
Estimating  Uncertainty  Guidelines”  [Anderson  and  Cherwonik  1997].  Additional 
sources  of  background  information  include  many  introductory  operations  analysis 
textbooks  and  the  users’  manuals  of  commercially  available  simulation  software.  Both 
the  @Risk  [Palisades  Corporation,  1997]  and  Crystal  Ball  [Decisioneering  Incorporated 
1997]  users’  manuals,  contain  thorough  discussion  on  performing  cost-risk  analysis. 

1.  Problems  Associated  With  Point  Estimation 

The  current  engineering  cost  models  estimate  cleanup  costs  with  a  single  point 
estimate.  Due  to  uncertainty,  it  is  highly  unlikely  that  any  point  estimates  are  exactly 
correct,  and  a  single  point  estimate  gives  the  decision-maker  no  information  on  the 
probable  range  of  cost  values.  Point  estimates  attempt  to  provide  a  single  “best 
estimate,”  but  these  point  estimates  suffer  from  ambiguity  in  what  the  “best  estimate” 
really  means.  Should  this  “best”  estimate  represent  the  “most  likely”  (mode)  cost,  the 
“average”  (mean)  cost  or  the  50th  percentile  (median)  cost? 
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Often,  cost  analysts  determine  total  program  costs  by  summing  up  the  “best 
estimate”  for  each  cost  element  of  a  program.  Cost  analysts  call  the  result  the  “roll-up” 
total  cost,  and  often  this  “roll-up”  is  statistically  incorrect  because  each  cost  element  of 
the  total  cost  has  uncertainty,  and  therefore  has  an  underlying  probability  distribution. 
Furthermore,  cost  analysts  may  report  some  of  these  “best  estimates”  as  the  mean  of  the 
underlying  distribution,  some  as  the  mode,  and  some  as  the  median.  Even  if  the  analyst 
reports  each  cost  element  in  the  same  manner  (e.g.,  all  modes),  summing  these  values 
may  not  give  the  intended  results.  When  the  analyst  sums  measures. of  central  tendency 
of  varying  probability  distributions,  very  rarely  is  the  result  the  matching  measure  of  the 
distribution  of  the  total.  The  reason  is  because  very  few  probability  distributions  having 
a  mean,  mode,  and  median  located  at  the  same  value.  The  triangular  distribution’s 
probability  density  function  (PDF)  illustrates  this  point.  The  peak  of  a  triangular 
distribution  is  the  “most-likely”  (mode)  cost,  and  the  end  points  are  the  extreme  low  and 
high  costs.  Figure  2  shows  a  triangular  distribution  of  a  cost  element  where  the  mode, 
mean,  and  median  are  not  collocated. 
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Figure  2:  The  Triangular  PDF.  The  triangular  PDF  is  an  example  of  a  distrihution 
where  the  mode,  mean  and  median  are  not  collocated. 

When  many  independent  cost  elements  are  summed  to  form  a  total  cost,  the 
central  limit  theorem  states  that  the  distribution  of  the  sum  should  have  an  approximately 
normal  distribution  with  a  mean  equal  to  the  sum  of  the  individual  means.  Since  the 
mean,  median,  and  mode  of  a  normal  distribution  are  all  equal  (located  at  the  mid  point  of 
the  distribution)  it  follows  that: 

1.  Total  cost  mean  =  Sum  of  cost  element  means. 

2.  Total  cost  mode  =  Sum  of  cost  element  means. 

3.  Total  cost  median  =  Sum  of  cost  element  means. 

And,  therefore,  it  is  most  often  the  case  that: 

1.  Total  cost  median  9^  Sum  of  cost  element  medians. 

2.  Total  cost  mode  9^  Sum  of  cost  element  modes. 

Often  in  point-estimates  the  “best  estimate”  is  the  mode  estimate,  but  Figure  3 
shows  a  common  example  of  when  the  sum  of  the  modes  of  the  cost  element  distributions 
does  not  equal  the  mode  of  the  total  cost.  In  this  case  the  “roll-up”  cost  lies  to  the  left  of 


the  total  cost  mode. 


Cost-Element 

Distributions 


Normal  Distribution  of  the  Sum  of  the  Cost-Elements 


Most  likely 


Figure  3:  Roll-up  Most  Likely  Cost  vs  Actual  Most  Likely  Cost.  The  figure  shows 
that  the  sum  of  the  cost  element  modes  is  often  not  equal  to  the  mode  of  the  total 
cost  distribution. 

For  most  distributions  the  sum  of  cost  elements’  most  likely  costs  does  not  equal  the  most 
likely  total-cost.  When  the  number  of  cost  elements  is  small,  there  is  no  expectation  that 
the  total-cost  distribution  would  be  normally  distributed'.  However,  it  is  still  unlikely  that 
the  sum  of  the  most  likely  costs  equals  the  most  likely  total  cost. 
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2.  Development  of  Total  Cost  Probability  Distributions 

In  DOD  acquisition  programs,  the  roll-up  method  tends  to  underestimate  costs  by 
a  wide  margin  [Book  1999].  The  DOD  is  therefore  shifting  to  a  method  known  as  “cost- 
risk  analysis.”  The  method  involves  developing  a  PDF  for  the  overall  cost.  Two 
approaches  exist  for  developing  the  total-cost  PDF.  The  “top  down”  approach  involves 
subjectively  determining  a  PDF  for  the  total  cost  while  the  “bottom-up”  approach  builds 
a  PDF  for  each  cost  element.  The  “bottom-up”  approach  is  usually  preferred  because  it  is 
often  much  easier  to  estimate  the  PDFs  for  the  individual  cost  elements  than  for  the  total 
cost. 

In  the  “bottom-up”  approach  the  cost  analyst  estimates  the  PDF  for  each  cost 
element.  For  example,  it  may  be  very  hard  for  a  cost  analyst  to  subjectively  determine 
the  PDF  for  the  total  cost  of  developing  a  new  fighter  aircraft.  Therefore,  the  analyst 
breaks  down  the  aircraft  into  all  of  its  constiment  parts.  In  military  acquisition,  this  is 
usually  done  in  the  work  breakdown  stmcture.  The  analyst  may  be  able  to  make  good 
estimates  on  the  PDFs  for  the  development  costs  of  the  engine,  avionics,  weapon  system, 
radar,  and  other  constituents.  Next,  a  computer  statistically  sums  the  individual  PDFs  by 
Monte  Carlo  sampling  to  develop  a  PDF  for  the  total  cost.  The  final  PDF  provides  the 
user  with  a  probabilistic  estimate,  provides  information  on  cost  percentiles,  and  provides 
the  means  to  produce  confidence  intervals  for  the  estimated  cost  [Book  1999]. 

The  first  step  in  a  “bottom-up”  cost-risk  analysis  is  to  develop  the  PDFs  for  each 
cost  element.  Often  in  cost  estimating  very  little  information  is  known  about  the  true 
distribution  of  cost;  however,  even  limited  information  can  aid  the  cost  analyst  in 
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estimating  a  parametric  distribution.  The  following  discussion  shows  the  most 
commonly  used  cost  estimating  distributions  and  how  basic  assumptions  about  costs  can 
be  used  for  choosing  a  PDF. 

There  are  three  qualities  that  make  a  probability  distribution  desirable  for 
modeling  cost:  the  distribution  contains  only  positive,  the  distribution  has  aright  skew, 
and  the  distribution  has  easily  estimated  parameters.  Cost  analysts  usually  desire 
distributions  with  non-negative  ranges  because  they  usually  define  costs  as  non-negative 
values.  A  right  skew  for  cost  distributions  is  a  result  of  market  place  economics.  Costs 
tend  to  be  located  around  the  mode,  but  very  high  costs  occur  with  small  probabilities. 
Unfortunately  for  the  consumer  very  low  costs  do  not  usually  happen  with  any 
probability.  Two  commonly  used  cost  distributions  that  have  these  properties  are  the 
Weibull  and  lognormal  distributions.  These  distributions  are  usually  parameterized  with 
two  parameters,  but  an  alternate  three-parameter  distribution  can  be  used  to  shift  the 
distributions  away  from  the  origin.  These  robust  distributions  can  take  many  shapes  and 
can  usually  be  developed  to  fit  right  skewed  cost  data. 
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Figure  4:  A  Right  Skewed  lognormal  Distribution.  Both  the  lognormal  distribution 
and  certain  Weibull  distributions  can  take  on  the  right  skewed  shape  as  seen  in  this 
figure.  Both  of  these  distributions  are  desirable  for  modeling  cost  data  because  their 
ranges  are  non-negative  and  the  right  skew  is  a  typical  phenomenon  of  costs. 

Although  the  Weibull  and  the  lognormal  distributions  tend  to  model  true  cost 
distributions  well,  their  parameters  are  not  easily  defined.  In  contrast,  cost  analysts 
sometimes  use  the  triangular  distribution  because  it  can  be  constructed  easily.  As 
previously  introduced,  this  distribution  requires  three  parameters:  an  absolute  low  cost,  a 
most  likely  cost,  and  an  absolute  high  cost.  Even  with  limited  prior  data  these  parameters 
can  be  determined  based  on  an  expert’s  opinion.  A  triangular  distribution  appears  in 
Figure  5. 


Figure  5:  Triangular  PDF  Construction.  Cost  analysts  often  use  the  triangular 
PDF  to  depict  the  distribution  of  a  cost-element  because  it  can  be  easily  created 
with  only  assumptions  of  the  low,  high,  and  most  likely  cost. 

Cost  analysts  also  commonly  use  the  normal  or  Gaussian  distribution.  The 

normal  distribution’s  familiar  symmetric  bell-shape  supports  the  assumption  that  the  cost 
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is  equally  likely  to  be  higher  or  lower  than  the  most  likely  cost.  The  normal  distribution 
is  a  two-parameter  distribution  that  is  completely  defined  by  its  mean  p.  and  its  standard 
deviation  a.  The  distribution  benefits  from  the  fact  that  its  mean,  mode,  and  median  all 
occur  at  the  same  point  centered  in  the  middle  of  the  distribution.  This  property 
simplifies  many  computations,  and  this  point  represents  the  “best  estimate”  of  the  cost 
under  any  definition  (mean,  mode,  or  median).  To  construct  the  distribution,  the  analyst 
often  estimates  the  value  that  is  greater  than  95%  off  all  possible  costs.  This  value  is 
roughly  1.96  standard  deviations  from  the  most  likely  cost,  and  the  analyst  uses  this  fact 
to  determine  the  value  of  the  standard  deviation.  With  this  value,  the  analyst  has  both 
parameters  for  defining  the  distribution.  The  normal  PDF  also  gains  favor  from  the  fact 
that  many  naturally  occurring  numerical  distributions  are  well  estimated  by  this  PDF. 
Figure  6  depicts  the  normal  PDF. 


Figure  6:  Normal  PDF  Construction.  Cost  analysts  often  use  the  normal  PDF  when 
there  is  an  assumption  that  the  cost  is  equally  likely  to  be  greater  than  or  less  than 
the  most  likely  cost.  The  mean,  median,  and  mode  are  all  located  at  the  same  point, 
which  simplifies  many  computations. 

The  third  distribution  that  cost  analysts  sometimes  use  is  the  uniform  distribution. 
This  distribution  does  not  have  a  single  most  likely  value,  and  all  outcomes  are  equally 
likely  to  occur  between  some  high  and  low  cost.  It  is  unlikely  that  the  uniform 
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distribution  models  the  trae  nature  of  a  cost  element,  and  is  only  used  when  there  is  no 
idea  about  the  relative  likelihood  of  possible  outcomes.  The  uniform  distribution  can  be 
modeled  with  two  parameters:  low  cost,  and  high  cost.  Figure  6  depicts  the  uniform 
PDF. 


Low  Cost  LBghCost 

Figure  7:  Uniform  PDF  Construction.  Cost  analyst  use  the  uniform  PDF  when 
there  is  no  knowledge  of  the  most  likely  cost,  but  it  is  assumed  that  the  costs  fall 
within  a  range  bounded  by  a  high  and  low  cost  with  equal  probability. 

In  the  second  step  of  a  “bottom-up”  risk  analysis,  one  random  sample  is  taken 
from  each  of  the  cost  elements  and  then  summed  together  to  form  a  single  sample  for  the 
total  cost  distribution.  The  Monte  Carlo  simulation  performs  this  sampling  numerous 
times  to  form  a  frequency  distribution  of  the  total  cost.  Figure  8  graphically  depicts  this 
procedure. 
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Figure  8:  Monte  Carlo  Simulation  to  Form  the  Total  Cost  PDF.  Monte  Carlo 
simulation  produces  a  total  cost  probability  distribution  from  individual  cost 
element  distributions. 

3.  Sampling  From  Historic  Data 

When  an  abundance  of  historic  cost  data  exists,  parameterized  PDFs  do  not  have 
to  be  hypothesized.  Instead  of  sampling  from  the  hypothesized  PDFs  of  each  cost 
element,  samples  from  the  empirical  data  set  can  be  taken.  To  demonstrate  how  this 
method  works,  assume  that  a  cost  analyst  uses  cost  data  from  the  cleanup  of  15  OOUs  to 
estimate  the  cost  of  a  proposed  cleanup.  Figure  9  presents  the  histogram  of  the  labor 
costs  for  the  15  selected  OOUs. 


29 


8  10  12  14  16  18  20  22  26 

Labor  Costs  in  Thousands  (CY99$/acre) 


Figure  9:  Histogram  ofLabor  Costs  for  15  OOUs.  Monte  Carlo  simulation  can  use 
empirical  cost  distributions  from  the  cost  element  instead  of  hypothesized  PDFs. 

The  method  of  sampling  from  historic  data  is  analogous  to  creating  a  spinner  that 
has  slices  proportional  to  the  frequency  of  each  value.  For  the  above  histogram  the 
spinner  looks  like  Figure  10. 

Sampling  Labor  Costs 
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Figure  10:  An  Analogy  for  Sampling  From  Empirical  Data.  A  spinner  models 
sampling  from  empirical  data  by  creating  slices  proportional  in  size  to  the  relative 
frequency  of  each  cost. 

By  this  analogy  a  person  draws  a  random  sample  by  spinning  the  spinner  and  recording 
the  value  under  the  spinner’s  pointer.  A  cost  structure  composed  of  N  cost  elements 
requires  N  separate  spinners.  For  each  sample  the  person  spins  the  N  spinners,  records  all 
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N  values,  and  sums  the  values  to  form  one  sample  for  the  total  cost.  The  person  performs 
this  procedure  many  times  to  form  a  histogram  of  the  total  cost. 

4.  Weighted  Estimate  of  a  Distribution  Function 

Both  methods  of  sampling,  from  historic  data  and  from  hypothesized  PDFs,  have 
advantages  and  disadvantages.  It  may  be  difficult  to  fit  cost  probabilities  with  a  common 
parameterized  distribution,  while  sampling  from  historic  information  does  not  require 
fitting  a  distribution.  Sampling  from  historic  data  provides  some  actual  information 
about  the  tme  cost  distribution.  However,  sampling  from  the  historic  data  suffers  when 
only  a  few  data  points  exist  because  sampling  purely  from  these  points  is  equivalent  to 
assuming  that  the  costs  can  only  take  on  one  of  these  few  values.  Most  likely  the  data  set 
does  not  contain  the  values  near  the  extreme  high  or  extreme  low  of  the  true  cost 
distribution.  Thus,  when  sampling  purely  from  the  historic  data,  a  lot  of  information  in 
the  tails  of  the  distribution  may  be.  The  ideal  probability  distribution  for  a  cost  element 
benefits  from  the  properties  of  both  a  hypothesized  parametric  probability  distribution 
and  an  empirical  data  set. 

The  benefits  of  both  methods  of  sampling  can  be  obtained  by  using  a  method 
similar  to  the  Bayesian  estimate  of  the  distribution  function  [Ferguson  1973,  p.272]. 
Ferguson  shows  that  the  Bayesian  estimate  of  a  probability  function  can  be  expressed  as 
the  weighted  average  of  the  hypothesized  prior  distribution  and  the  empirical  distribution 
of  the  data.  For  a  data  set  of  n  observations  the  Bayesian  estimate  is 

=  +  (4.1) 

Where: 
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p„=ala+n  (4.2) 

/(,(/)  -  The  prior  hypothesized  PDF 

/„(t  I  -  The  empirical  PDF  given  the  data  set  Xi,...,Xn 

a  -  Weighting  factor 

Equation  (4.1)  weights  the  prior  distribution  by  pn  and  the  empirical  distribution 
by  (l-/7„ ).  It  can  be  seen  that  as  n  increase  pn  decreases;  hence,  the  equation  weights  the 
prior  distribution  less  and  weights  the  empirical  distribution  more.  The  constant  a  is  a 
weighting  factor  that  can  be  interpreted  as  the  faith  in  the  prior  distribution  measured  in  n 
units.  This  thesis  uses  this  weighted  distribution  to  estimate  cost  element  probability 
distributions. 

5.  Handling  Correlation 

The  traditional  Monte  Carlo  sampling  theory  of  summing  cost  elements  to 
produce  a  total  cost  probability  distribution  is  valid  only  when  the  cost  elements  are 
independent.  It  is  often  the  case  that  cost  elements  are  not  independent  but  correlated. 
Two  random  variables  X  and  Y  are  correlated  when  the  value  of  X  has  some  predictive 
power  on  the  value  of  Y.  For  example  labor  costs  and  material  costs  in  UXO  cleanups 
have  an  intuitive  relationship  that  leads  to  correlation.  A  difficult  cleanup  that  uses  a  lot 
of  materials  requires  a  lot  of  labor  hours  to  employ  the  materials;  hence,  a  strong  positive 
correlation  exists.  Traditional  Monte  Carlo  sampling  provides  the  correct  value  for  the 
total  cost  mean  regardless  of  whether  the  cost  elements  are  independent  or  correlated; 
however,  the  variance  of  the  total  cost  probability  distribution  depends  strongly  on  the 
correlation  of  the  individual  cost  elements  [Book  1999].  Let  Xi,  X2,  ...X„  be  n  random 
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variables  representing  n  cost  elements,  //,  the  mean  of  cost  element  X/,  <7,  the  standard 
deviation  of  cost  element  X,,  and  TC  be  the  total  cost.  It  follows  that: 

n 

Total  Cost:  TC  =  X  ^ «  (4-3) 


Mean  of  Total  Cost  =  E 


Ex,  =Xa 


i=l 


J 


1=1 


Variance  of  Total  Cost  =  var 


Sx, 

V'=‘  y 


(4.4) 

(4.5) 


n  n  7“1 

=  E®’" + (4-6) 

,=1  y=2  /=1 

As  seen  in  equation  (4.4)  no  cost  element  correlation  term  appears  in  the 
calculation  of  the  total  cost  mean;  however,  the  correlation  coefficient,  pij,  appears  in 
equation  (4.6),  the  calculation  of  the  total  cost  variance.  Not  accounting  for  correlation  is 
equivalent  to  setting  />y  equal  to  zero,  and  equation  (4.6)  then  simplifies  to: 

n 

Var  of  Total  Cost  =  X  (4.7) 

i=l 


The  differences  in  equation  (4.7)  and  (4.6)  shows  that  strong  positive  correlation 
can  greatly  increase  the  total  cost  variance  while  strong  negative  correlation  can  greatly 
reduce  the  total  cost  variance.  Thus  not  accounting  for  correlation  gives  the  incorrect 
total  cost  probability  distribution.  Accounting  for  correlation  is  not  a  simple  proposition: 
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The  ideal  way  to  simulate  total  cost  would  be  to  specify  inter-cost- 
element  correlations,  generate  correlated  random  numbers,  and  sum  them. 
Probability  theory  allows  this  to  be  done  exactly  only  for  multivariate 
normal  cost  distributions,  not  in  general  [Book  1999] 

Book  further  points  out  that  Lurie  and  Goldberg  [1998]  have  been  working  for 
several  years  on  the  correct  method  to  generate  correlated  random  numbers  that  correctly 
model  cost-element  cost  distributions. 

An  accepted  approximate  solution  to  the  above  problem  is  to  use  rank-order 
correlation,  a  distribution  free  measure  of  correlation.  Several  rank-order  measures  of 
correlation  exist  including  Kendall’s  tau  and  Spearman’s  rho.  The  two  most  popular 
software  packages  that  perform  cost-risk  analysis:  Crystal  Ball  [Decisioneering 
Incorporated,  1998]  and  @  Risk  [Palisades  Corporation  1997],  use  rank-order  correlation. 
Rank-order  correlation  between  a  pair  of  data  sets  involves  ranking  the  corresponding 
sets  from  lowest  to  highest  value.  To  illustrate  the  calculation  of  Spearman’s  rho.  Table 
4  shows  the  labor  cost  per  acre  and  the  corresponding  material  cost  per  acre  for  10 
OOUs.  Next  to  each  cost  is  the  rank  of  the  cost  within  its  column. 
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Material/Acre 

Material 
Rank  R(Yi) 

$30.05 

1 

$2,254.77 

10 

$153.52 

3 

$243.56 

6 

$113.93 

2 

$175.19 

4 

$208.83 

5 

$312.65 

8 

9 

7 

oou 

Labor/Acre 

Cost 

Labor 
Rank  R(Xi) 

1 

$360.20 

1 

2 

$532.24 

2 

3 

$705.21 

3 

4 

$809.14 

4 

5 

$1,071.02 

5 

6 

$1,196.95 

6 

7 

$1,402.56 

7 

8 

$2,445.69 

8 

9 

$3,068.89 

9 

10 

$4,225.71 

10 

Table  4:  Determining  Spearman’s  Rho  by  Rank  Ordering  Cost  Elements.  This 
example  shows  how  to  determine  rank*order  correlation  between  variable  X  (Labor 
cost)  and  variable  Y  (Material  Cost). 

K  the  cost  elements  have  perfect  positive  rank-correlation  (p  =  +1)  then  the 
corresponding  rank  value  for  each  pair  would  be  equal.  If  the  cost  elements  have  perfect 
negative  correlation  (p  =  -1),  then  the  lowest  labor  costs  would  be  paired  with  the  highest 
material  costs,  and  the  highest  labor  costs  would  be  paired  with  the  lowest  material  costs. 


Equation  (4.8)  calculates  p  [Conover  1999,  p.  315]: 


X  R{X  ,)RiY,)  -  n 


n  +  1 


(4.8) 


X  R{X  -  n 


n  +  1 


2  'N  9 


X  R(Y,)^  -  n 


f  n  +  1 


.2 


Where: 


R(Xi)  -  Rank  of  the  ith  X  variable 
R(Yi)  -  Rank  of  the  ith  Y  variable 


For  the  above  example  (X  =  labor  cost,  Y  =  material  cost,  n  =  10),  p  is  calculated 
as  follows: 
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338-302.5 


(385  -  302.5)1  (385  -  302.5)l 


=  .430 


(4.9) 


The  process  of  generating  rank-correlated  pairs  of  sampled  values  is  a  two  step 
process  that  requires  the  user  to  supply  correlation  coefficients  for  each  pair  of  data.  In 
the  first  step,  the  simulation  software  generates  randomly  distributed  rank  scores  for  each 
random  variable.  The  software  generates  one  score  for  each  iteration  of  the  simulation. 
Sampling  10,000  times  would  require  10,000  random  rank  scores  for  each  random 
variable.  These  rank  scores  are  then  rearranged  to  give  pairs  of  scores  that  match  the 
desired  rank  correlation  coefficient.  In  the  second  step,  the  software  generates  another 
set  of  random  numbers  for  each  random  variable,  and  again  generates  one  number  for 
each  iteration  of  the  simulation.  For  each  variable  the  procedure  assigns  the  smallest 
random  number  to  the  smallest  rank  score,  the  second  smallest  random  number  to  the 
second  smallest  rank  score,  and  so  on.  The  final  result  is  a  set  of  pairs  of  random 
numbers  that  can  be  used  in  sampling  values  from  correlated  distributions  [@Risk  User’s 
Manual,  p.  179].  The  software  creates  correlated  pairs  for  every  random  variable  with 
every  other  random  variable  and  finally  arranges  the  pairs  to  fit  a  mapix  of  specified 
correlation  coefficients. 
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V.  MODEL  APPLICATION 


This  thesis  creates  a  probability  distribution  of  the  CTC  of  a  proposed 
contaminated  area  by  analyzing  cost  data  from  previous  cleanups.  Location 
characteristics,  ordnance  characteristics,  and  cleanup  requirements  can  be  vastly  different 
for  two  separate  projects.  In  order  to  account  for  variance  in  costs  caused  by  these 
differences,  the  cost-risk  model  uses  only  data  from  areas  “similar”  to  the  proposed  area 
of  cleanup.  Once  similar  records  are  found  the  cost-risk  model  uses  the  historic  cost  data 
to  generate  cost  per  acre  estimates  that  incorporate  cost-risk  analysis. 

As  previously  mentioned,  USAESC,  Huntsville  maintains  the  Ordnance  and 
Explosives  Database  of  completed  and  ongoing  UXO  removal  projects.  Although  this 
database  is  sparsely  populated  with  cost  information,  there  is  interest  within  the  DOD  to 
produce  and  maintain  a  DOD-wide  ordnance  and  explosive  management  system.  The 
assumption  of  this  thesis  is  that  either  the  OED  will  be  populated  in  the  future  or  another 
repository  of  information  will  be  created.  This  thesis  presents  a  model  with  designs  to 
incorporate  the  OED  data  fields,  but  the  model  can  be  easily  modified  to  accept  historic 
information  from  any  repository  of  UXO  data.  Appendix  A  outlines  the  information 
fields  needed  by  the  cost-risk  model  but  that  currently  do  not  exist  in  the  OED.  The  same 
appendix  also  makes  recommendations  to  modify  some  fields  so  that  the  cost-risk  model 
can  incorporate  them. 
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A. 


DATAHELDS 


The  following  discussion  outlines  the  necessary  fields  that  the  cost-risk  model 
must  obtain  from  a  database.  Although  these  recommendations  have  designs  to 
specifically  incorporate  the  OED,  the  same  recommendations  hold  for  any  other  UXO 
data  repository  if  developed.  The  cost-risk  model  uses  database  fields  segregated  into 
three  groups:  record  identification,  OOU  descriptive  characteristics,  and  cost  information. 
The  record  identification  fields  provide  a  means  to  identify  each  OOU  as  unique.  The 
fields  include  site,  project,  and  OOU  identifiers.  Table  5  shows  the  required  record 
identification  fields. 


1  Field  Name 

Description 

Name  of  site  (e.g..  Fort  Ord,  Aberdeen  Proving  Grounds). 

PROJECT.NO 

Unique  Identifier  throughout  the  database.  Identifies  the 
project  that  an  OOU  falls  under. 

OOU_NO 

Unique  identifier  within  each  project.  Identifies  a  specific 

OOU  within  a  project. 

Table  5:  Record  Identification  Fields.  The  cost-risk  model  needs  record 
identification  fields  to  uniquely  identify  OOUs.  All  fields  exist  in  the  OED. 

The  second  group  of  fields  is  the  OOU  descriptive  characteristics.  These  fields 
include  environmental  characteristics,  ordnance  characteristics,  and  cleanup 
characteristics.  These  are  the  fields  that  the  user  queries  to  find  records  similar  to  the 
proposed  cleanup.  Table  6  summarizes  the  OOU  descriptive  characteristic  fields  required 
by  the  cost-risk  model. 
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Field  Name 

Description 

TYPE’ 

Type  of  cleanup  (chemical  weapons  or  conventional) 

ACRES’ 

Total  OOU  acres 

SOIL’ 

Difficulty  factor  based  on  the  type  of  soil  in  the  OOU 

TERRAIN’ 

Difficulty  factor  based  on  the  dominant  terrain  in  the  OOU 

VEGETATION” 

Difficulty  factor  based  on  the  dominant  vegetation  in  OOU 

DENSITY  ” 

Density  of  ordnance  within  the  OOU 

DEPTH_CLEAR'’ 

Depth  of  clearance  in  feet 

TYPE.ORDNANCE” 

A  weighted  difficulty  factor  based  on  the  types  of  ordnance 
contained  in  the  OOU 

SURVEY_TYPE‘ 

Method  of  surveying  (mag  and  flag  operation,  geophysical) 

Table  6:  Ordnance  Operable  Unit  (OOU)  Descriptive  Characteristics.  The  cost- 
risk  model  requires  these  flelds  to  identify  an  OOUs  environmental  characteristics, 
its  unexploded  ordnance  characteristics,  and  its  cleanup  characteristics. 

’Field  currently  exists  in  OED 

’’Recommend  modification  of  existing  field  in  OED  (see  Appendix  A) 

‘Recommend  addition  of  field  in  the  OED  (see  Appendix  A) 

The  final  required  fields  from  the  database  are  the  cost  information  fields.  The 

cost  fields  in  the  database  summarize  information  on  six  cost  elements  that  make  up  total 

cost.  These  elements  are: 

1.  EECA  Labor  Cost  per  acre. 

2.  EECA  Material  Cost  per  acre. 

3.  EECA  Travel/Per  Diem  Cost  per  acre. 

4.  Removal  Action  Labor  Costs  per  acre. 

5.  Removal  Action  Material  Cost  per  acre. 

6.  Removal  Action  Travel/Per  Diem  per  acre. 

The  model  converts  all  costs  to  FY99  costs.  The  cost-risk  model  produces  estimates 
following  a  completed  ASR;  therefore,  it  does  not  include  cost  information  on  the  first 
two  phases  of  the  remediation  process  (Preliminary  Assessment  and  Investigation  (ASR) 
phases).  The  cost  fields  that  the  cost-risk  model  obtain  from  the  database  differ  from  the 
fields  in  the  current  OED  structure.  The  recommended  fields,  however,  reflect  data  that 
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is  realistically  accessible.  Ideally  cost  data  would  exist  for  all  phases  of  a  project 
(Preliminary  Assessment,  Investigation,  and  Removal  Action);  unfortunately  this  thesis  is 
only  able  to  acquire  Removal  Action  cost  data.  Huntsville  Center  cost  engineers  estimate 
Removal  Action  costs  account  for  80%  of  the  total  cost  to  complete  [Young  1999].  For 
the  purpose  of  demonstrating  this  model,  the  thesis  assumes  the  costs  for  the  three  EECA 
cost  elements  to  be  equal  to  two  percent  of  its  corresponding  RA  cost  element.  The  basis 
of  this  rough  figure  is  the  current  procedure  of  sampling  approximately  two  percent  of  an 
OOU  during  the  EECA  phase  in  order  to  determine  ordnance  density.  Table  7  outlines 
the  cost  fields  that  the  cost-risk  model  uses. 


Field  Name 

Description 

EECA_YEAR‘ 

Year  EE/C  A  completed. 

RA_YEAR‘ 

Year  RA  completed 

EECA_LABOR_COST‘’ 

EE/CA  Labor  Cost 

EECA_MAT_COST'’ 

EE/CA  Material  Costs 

EECA_TRAVPDIEM_COST'’ 

EE/CA  Travel  Per  Diem  Costs 

RA_LABOR_COST'’ 

RA  Labor  Cost  for  the  OOU 

RA.MAT.COST” 

RA  Material  Cost  for  the  OOU 

RA.TRAVPDffiM.COST" 

RA  Travel  and  Per  Diem  cost  for  OOU 

Table  7:  Ordnance  Operable  Unit  (OOU)  Cost  Information.  These  fields  are 
needed  to  report  cost  information. 

“Field  currently  exists  in  OED 

‘’Recommend  modification  of  existing  field  in  OED  (see  Appendix  A) 

‘Recommend  addition  of  field  to  OED  (see  Appendix  A) 

B.  FINDING  SIMILAR  CLEANUPS 

The  user  imports  the  current  ordnance  data  file  into  the  cost-risk  model,  and  then 
begins  developing  a  cost-risk  estimate.  If  the  ASR  is  complete  for  a  site,  the  user  has 
approximate  information  on  the  areas  to  be  cleaned  up.  The  user’s  first  step  is  to  find 
records  similar  to  a  proposed  area  of  cleanup  by  filtering  the  data  based  on  the  OOU 
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descriptive  characteristics.  An  important  trade-off  exists  when  filtering;  as  the  user 
specifies  more  criteria  the  amount  of  available  records  decreases.  Conversely,  the  more 
detailed  the  search  criteria  the  more  similar  the  selected  OOUs  are  to  the  proposed 
cleanup  area.  Unfortunately  a  detailed  search  may  leave  few  records  for  sampling.  Since 
the  current  data  set  is  small  the  user  is  only  able  to  filter  the  data  using  a  few  different 
criteria.  As  the  database  grows  in  number  of  cleanups,  the  user  can  increase  the  search 
criteria. 

C.  FILTERING  PRECEDENCE 

The  user  filters  the  records  beginning  with  the  OOU  descriptive  field  that  is  most 
indicative  of  cost.  If  after  a  query  enough  records  remain,  the  user  refines  the  search  by 
querying  on  the  next  most  influential  field  and  so  on.  The  user  stops  filtering  when  he 
determines  that  further  filtering  could  reduce  the  data  set  below  a  useable  size.  The  OOU 
descriptive  fields  can  be  subdivided  into  three  groups  based  on  how  strongly  they 
influence  cost.  These  groups  consist  of  “type  of  cleanup,”  “primary  indicators,”  and 
“secondary  indicators.” 

1.  Type  of  Cleanup 

The  TYPE  field  determines  if  the  cleanup  is  a  chemical  weapons  cleanup  or  a 
conventional  weapons  cleanup.  Users  should  always  filter  this  field  first  since  chemical 
weapon  cleanups  are  far  more  difficult  than  conventional  weapons,  and  costs  of  chemical 
weapons  cleanups  are  not  indicative  of  the  costs  of  conventional  cleanups. 
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2.  Primary  Indicators 

Plotting  each  OOUs  descriptive  characteristic  versus  the  total  cost  for  RA 
determines  the  remaining  filtering  precedence.  Primary  fields  are  those  fields  that 
indicate  a  strong  relationship  between  them  and  costs.  The  first  plot  shows  density  of 
each  OOU  plotted  against  RA  total  cost.  Although  there  are  several  possible  measures  to 
describe  the  density  of  ordnance  within  an  OOU,  the  cost-risk  model  uses  total  scrap 
(UXO  scrap  and  non-UXO  scrap)  as  the  measure  of  density.  Density  usually  refers  to 
the  amount  of  UXO  per  acre,  but  this  definition  may  not  be  a  good  indicator  of  costs. 
Ordnance  technicians  must  remove  UXO  scrap  as  well  as  non-UXO  scrap  from  a 
contaminated  area,  and  they  still  must  excavate  false  anomalies  caused  by  non-UXO 
along  with  actual  UXO.  The  relationship  between  actual  UXO  density  and  non-UXO 
scrap  density  and  its  effect  on  costs  is  extremely  difficult  to  define.  This  thesis  defines 
density  as  the  total  scrap  metal  removed  from  an  area.  A  more  detailed  analysis  of  the 
relationship  of  UXO  scrap  and  non-UXO  scrap  and  its  effects  on  cost  is  beyond  the  scope 
of  this  thesis  and  should  be  the  subject  of  further  research  (see  Appendix  A  for  further 
discussion).  Figure  1 1  shows  the  relationship  of  density  to  cost.  This  plot  shows  a  strong 
relationship  of  increasing  costs  with  increasing  density. 
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Figure  11:  Scatter  Plot  of  OOU  Ordnance  Density  vs  Total  Cost.  The  scatterplot 
shows  a  strong  relationship  of  increasing  RA  costs  with  increasing  density.  The  log 
trend  line  suggests  that  costs  increase  proportionally  to  the  log  of  density. 

Figure  12  shows  that  the  increase  in  cost  is  roughly  linear  with  the  log  of  density. 


Figure  12:  Scatter  Plot  of  OOU  Ordnance  Log-density  vs  Total  Cost.  The  figure 
shows  a  strong  relationship  of  increasing  RA  costs  with  increasing  log-density.  The 
trend  line  shows  that  the  relationship  is  approximately  linear. 

The  relationship  of  depth  of  clearance  to  cost  appears  weaker  than  that  between 
density  and  cost,  but  the  plot  shows  that  as  depth  of  clearance  increases  total  cost  tends  to 
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increase.  Unfortunately  the  data  set  does  not  contain  any  clearances  in  excess  of  four 
feet.  Figure  13  plots  of  depth  of  clearance  versus  cost  and  adds  a  linear  trend  line  to 
show  the  increasing  relationship. 


Figure  13:  Scatter  Plot  of  OOU  Depth  of  Clearance  vs  Total  Cost.  The  figure  shows 
a  relationship  of  increasing  RA  costs  with  increasing  depth  of  clearance. 

A  similar  relationship  appears  in  the  plot  of  vegetation  difficulty  factor  versus 
total  cost  (see  Appendix  A  for  discussion  of  difficulty  factors).  In  Figure  14  the  reader 
sees  positive  correlation  between  total  cost  and  the  vegetation  factor  of  an  OOU.  As  the 
difficulty  factor  increases  the  cost  of  ordnance  removal  tends  to  increase. 
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Figure  14:  Scatter  Plot  of  OOU  Vegetation  Difficulty  Factor  vs  Total  Cost.  The 
figure  shows  a  relationship  of  increasing  RA  costs  with  increasing  vegetation 
difficulty  factor. 

2.  Secondary  Indicators 

Appendix  B  contains  the  scatter  plots  of  the  remaining  OOU  characteristics: 
topography,  soil  type,  and  ordnance  type.  These  plots  do  not  indicate  the  expected  cost 
relationships.  The  author  hypothesizes  that  these  secondary  fields  do  not  provide  the 
expected  cost  relationships  because  the  primary  indicators  overshadow  the  effects  of  the 
secondary  indicators.  For  example,  clay  soil  is  difficult  because  clay  contains  a  lot  of 
minerals  that  make  detection  of  UXO  difficult  using  conventional  magnetometers.  Most 
research  assumes  that  if  two  cleanups  are  identical  except  for  soil  type,  the  area  with  the 
higher  soil  type  difficulty  factor  will  be  more  expensive.  Unfortunately,  the  only  data 
points  with  clay  soils  also  have  low  densities  and  low  vegetation  factors,  and  thus  have 
low  costs.  The  second  explanation  for  why  secondary  indicator  cost  relationships  do  not 
reflect  expected  results  is  interactions  with  the  primary  fields.  For  example,  one  can 
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hypothesize  that  heavily  used  firing  areas  tend  to  be  located  in  flatter  terrain,  or  that 
certain  soil  types  indicate  the  amount  of  sustainable  overgrowth.  Analysis  with 
additional  data  should  better  reveal  the  true  cost  relationships  of  these  factors. 

D.  MODEL  INTRODUCTION 

1.  Initial  Worksheets 

Visual  Basic  for  Applications  creates  the  cost-risk  model  as  an  add-in  for 
Microsoft  Excel.  When  the  user  loads  the  add-in  into  Excel,  one  worksheet  is  visible  and 
two  worksheets  are  hidden.  The  hidden  worksheet  “InflationIndices”  contains  the 
inflation  indices  needed  to  normalize  all  costs  to  base  year  1999.  The  other  hidden 
worksheet  is  ‘Tables”  which  provides  non-changing  descriptive  information  about  the 
OOU  characteristics.  The  user  should  not  need  to  alter  or  access  either  of  these 
worksheets. 

When  the  model  user  opens  the  add-in  the  “Master”  worksheet  is  the  only  visible 
worksheet.  This  worksheet  contains  all  of  the  Record  Identification,  OOU  Descriptive 
Characteristics,  and  OOU  Cost  Information  fields,  as  discussed  in  subsection  V.A.2.  The 
add-in  can  update  these  fields  from  a  text  file  of  current  data  and  automatically  convert 
costs  to  fiscal  year  1997  costs.  Figure  15  shows  the  cost-risk  model  when  initially 
opened. 
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Figure  15:  View  of  Cost-Risk  Model  on  Opening.  This  figure  shows  the  only  visible 
worksheet  when  the  Excel  add-in  is  loaded.  Visible  in  the  figure  is  the  OOU  number 
and  associated  OOU  characteristics.  The  model  updates  this  worksheet  from  a  text 
file  of  current  data. 

2.  Forms 

Additionally,  the  add-in  contains  several  forms  that  the  user  can  use  to  input 
filtering  criteria.  The  scenario  discussion,  later  in  this  thesis,  presents  each  of  these  forms 
in  detail.  These  forms  provide  a  Graphical  User  Interface  and  simplify  navigation 
through  the  model.  Figure  16  shows  one  of  the  forms  that  assists  the  user  in  filtering 
records.  This  form  is  the  Depth  of  Removal  Form  used  to  filter  the  records  based  on  a 
minimum  and  maximum  depth  of  clearance. 
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Figure  16:  Example  of  a  Form.  The  Depth  of  Removal  Form  is  an  example  of  the 
many  forms  available  to  assist  the  user  in  entering  data.  The  user  can  use  the  scroll 
bars  at  the  bottom  to  set  the  minimum  and  maximum  depth  of  clearance.  The 
buttons  in  the  upper  right  help  to  navigate  through  the  model. 


3.  Menus 

The  add-in  creates  two  additional  menus  in  Excel.  “Cost  Forms”  and 
“Cost_ Analysis.”  These  two  additional  menus  appear  on  the  right  of  the  toolbar. 


JtW3b.lD^a,.,5«tK4w  ' CostfOrm^ ■  Gost_A^sfe' 

Figure  17:  Added  Excel  Menus.  The  cost  analysis  add-in  creates  two  additional 
menus  in  excel.  These  menus  are  “CostForms”  and  “Cost_Analysis,”  and  are 
visible  on  the  far  right  of  the  menu  bar. 


The  Cost  Forms  menu  provides  additional  means  to  navigate  through  the  different 


forms;  this  menu  lists  all  available  forms.  Figure  18  shows  this  pull  down  menu. 


nc  i  OR  HR 


Figure  18:  The  CostForms  Menu.  The  CostForms  menu  assists  the  user  in 
navigating  through  the  availahle  forms. 

The  “Cost_Analysis”  pull  down  menu  contains  submenus  that  allows  the  user  to  access 
the  model’s  important  functions.  These  three  functions  are:  1)  Filter  the  records  to  find 
similar  cleanups,  2)  Perform  an  analysis  on  the  filtered  historic  data,  and  3)  Build 
probability  density  functions  for  input  into  the  simulation  software.  The  scenario 
presentation  discusses  each  function  in  detail,  and  Figure  19  shows  the  “Cost_Analysis” 


menu. 


Figure  19:  The  “Cost_Analysis”  Menu.  The  “Cost_Analysis”  menu  provides  access 
to  the  model’s  important  functionality  including  filtering  records,  analyzing  historic 
costs,  and  building  probability  density  functions. 
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E.  SCENARIO  DEVELOPMENT  AND  MODEL  DEMONSTRATION 


This  thesis  uses  the  following  scenario  to  demonstrate  how  to  interface  with  the 
cost-risk  model  and  to  aid  the  user  in  the  understanding  of  the  internal  workings  of  the 
model. 


Scenario:  The  Marine  Corps  conducts  an  Archive  Search  Report  on 
Marine  Corps  Base  Camp  Lejeune,  North  Carolina.  The  ASR  shows  a 
suspected  Ordnance  Operable  Unit  of  45  acres  contaminated  with  a 
mixture  of  high  explosive  ordnance  including  155mm  Artillery  rounds, 

81mm  and  60mm  mortar  rounds,  40mm  grenades,  and  hand  grenades. 

The  ASR  does  not  suspect  any  small  arms  ammunition.  The  area  has  not 
been  used  for  several  years  and  the  vegetation  is  overgrown  with  shrubs 
and  small  trees.  The  contaminated  site  is  in  the  western  part  of  the  base 
with  predominately  flat  terrain.  The  soil  type  is  a  mixture  of  silt  and  sand. 

EOD  units  conduct  semiannual  range  sweeps  of  this  area,  and  only  a  small 
amount  of  surface  scrap  is  found  during  the  ASR.  Further,  range  control 
records  show  that  range  usage  for  this  area  is  low  because  of  its  proximity 
to  the  edge  of  the  base  property.  The  Marine  Corps  plans  to  convert  this 
area  back  to  a  maneuver  area. 

1.  Step  One:  Find  Similar  Cleanups  by  Filtering  Database 

Using  the  “Cost_Analysis”  menu  the  user  selects  the  sub  menu  “GUI  Search” 
(Graphical  User  Interface  Search)  or  “Manual  Search.”  The  GUI  Search  submenu  starts  a 
sequence  of  user  forms  that  help  refine  the  search  criteria.  The  Manual  Search  submenu 
sends  the  user  to  the  Master  worksheet  where  the  user  can  search  the  records  using 
Excel' s  AutoFilter  function. 

a.  Filter  by  Type  of  Cleanup 

If  the  user  selects  “GUI  Search”  the  model  first  presents  the  Type  of 
Cleanup  Form  where  the  user  checks  what  type  of  cleanup  concerns  them:  conventional 
ordnance  and  explosives  or  chemical  weapons.  The  ASR  for  the  scenario  shows  it  is  a 
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conventional  cleanup,  and  therefore  the  user  selects  “Ordnance  and  Explosives”  as  Figure 
20  shows. 


Figure  20:  The  Type  of  Cleanup  Form.  This  form  helps  to  filter  the  database  based 
on  the  type  of  cleanup.  When  the  user  selects  the  Ordnance  and  Explosives  Button 
the  model  filters  out  all  of  the  chemical  weapons  cleanups. 

By  selecting  the  “Next»”  Button  the  model  presents  the  user  with  the 
next  filtering  form;  the  Close  and  Navigate  buttons  close  out  the  form  and  allow  the  user 
to  go  to  different  forms  or  worksheets.  Finally,  the  Filter  Button  shows  the  user  how 
many  records  are  remaining  in  the  filtered  set.  By  selecting  “Ordnance  and  Explosives” 
and  clicking  the  Filter  Button,  the  user  reduces  the  data  set  to  nineteen  records  (i.e.,  the 
current  data  base  does  not  contain  any  chemical  weapons  cleanups).  Figure  21  shows 
the  Found  Records  Form  that  displays  the  number  of  remaining  records. 


51 


Figure  21:  The  Found  Records  Form.  The  user  can  use  the  Found  Records  Form 
from  any  location  in  the  model  to  show  how  many  records  remain  in  the  selected 
data  set. 


b.  Filter  by  Density 

The  “Next»”  Button  sends  the  user  to  the  Density  Form.  Density  tends 
to  be  a  strong  predictor  of  cost,  unfortunately  prior  to  surveying  the  user  may  only  be 
able  to  make  a  rough  guess  on  the  expected  density.  The  user  should  use  a  broad  range 
if  there  is  uncertainty  concerning  the  density.  The  form  provides  four  preset  ranges: 
“Sparse,”  “Low,”  “Medium,”  and  “High.”  Dividing  the  current  data  set  into 
approximately  equal  quartiles,  determines  these  preset  ranges.  For  the  given  scenario,  the 
ASR  shows  that  the  Marine  Corps  did  conduct  periodic  range  sweeps  and  that 
investigators  found  only  a  small  amount  of  surface  scrap  during  the  site  investigation. 

For  these  reasons  the  user  is  somewhat  confident  that  the  range  is  sparsely  populated  and 
selects  a  liberal  density  (total  scrap  per  acre)  range  of  0-100  pounds  per  acre.  This 
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selection  reduces  the  data  set  to  14  records.  The  Density  Form  appears  below  in  Figure 


22. 


Figure  22:  The  Density  Form.  The  Density  Form  allows  the  user  to  filter  the  data 
set  based  on  the  total  scrap.  Prior  to  performing  an  EECA  the  user  may  only  have  a 
rough  estimate  on  the  amount  of  scrap  per  acre.  The  user  should  enter  a  liberal 
density  range  unless  surveying  data  gives  a  good  estimate  on  the  scrap  density. 

c.  Filter  by  Vegetation 

The  next  form  in  the  filtering  precedence  is  the  Vegetation  Form.  An 
ASR  categorizes  an  OOUs  vegetation  factor  based  on  the  primary  vegetation  throughout 
the  OOU.  The  data  set  contains  a  ranking  for  each  OOU  based  on  vegetation  difficulty 
factors.  The  cost-risk  model  adapts  the  RACER/ORCA  difficulty  factors,  and  these 
factors  appear  in  Table  8.  For  simplicity,  the  cost-risk  model  refers  to  these  rankings  as 
VI  through  V5. 
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Table  8:  Vegetation  Characteristics  and  Associated  Difficulty  Factors.  In 
accordance  with  the  RACER/ORCA  model,  the  cost-risk  model  ranks  vegetation  by 
difficulty  factors.  Unlike  density,  the  user  has  certainty  of  the  type  of  vegetation 
contained  in  a  contaminated  area. 


The  contaminated  area  in  the  scenario  contains  heavy  vegetatioii;  therefore,  the  user  may 


choose  to  select  categories  V4  and  V5.  This  selection  narrows  the  data  set  down  to  1 1 


Figure  23:  The  Vegetation  Form.  The  model  user  can  use  this  form  to  set  the 
minimum  and  maximum  vegetation  type  for  the  contaminated  area.  For  the 
current  scenario  a  range  from  V4  to  V5  best  depicts  the  area. 
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d.  Filter  by  Depth  of  Clearance 

Following  the  Vegetation  Form  is  the  Depth  of  Removal  Form.  The  final 


depth  of  clearance  requires  deliberation  among  all  concerned  parties  prior  to  RA,  but  the 
intended  future  use  of  the  land  suggests  about  a  four-foot  clearance.  Setting  the 
minimum  depth  to  two  feet  and  the  maximum  depth  to  six  feet  reduces  the  number  of 
available  records  to  nine.  The  Depth  of  Removal  Form  appears  in  the  Figure  24. 


Figure  24:  The  Depth  of  Removal  Form.  The  Depth  of  Removal  form  allows  the 
user  to  And  historic  cleanups  with  similar  clearance  depths.  For  the  current 
scenario,  Altering  by  depth  reduces  the  amount  of  available  records  to  nine. 

e.  Filter  by  Topography 

Once  the  user  completes  filtering  on  the  primary  cost  indicators  he  begins 


filtering  with  the  secondary  cost  indicators.  Due  to  small  number  of  records  in  the 
filtered  data  set  (9  records),  the  user  would  likely  quit  at  this  point.  When  more  records 
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are  added  to  the  database,  the  user  would  likely  have  enough  remaining  records  to 
continue  refining  the  search.  However,  in  this  case  the  user  should  concentrate  on 
removing  the  extremes  of  the  secondary  indicators  so  as  not  to  significantly  reduce  the 
number  of  records  any  further.  Like  vegetation  the  model  ranks  topography  using  the 
RACER/ORCA  difficulty  factors.  Table  9  lists  the  topography  categories  and  the 
associated  difficulty  factors.  The  cost-risk  model  labels  these  categories  as  T1  through 
T5. 


Topography 

Difficulty  Factor 

Tl:  Flat 

0 

T2:  Gently  Rolling 

0.1 

T3:  Heavy  Rolling 

0.3 

T3:  Flat  with  Gorges 

0.3 

T4:  Rolling  With  Gorges 

0.5 

T5:  Mountainous 

0.7 

Table  9:  Topography  Categories  and  Associated  Difficulty  Factors.  In 
Accordance  with  the  RACER/ORCA  model,  the  cost-risk  model  ranks  topography 
by  difficulty  factors. 

The  contaminated  area  in  the  scenario  falls  in  the  T1  and  T2  categories,  by  selecting  a 
minimum  of  T1  and  a  maximum  of  T4  the  user  eliminates  one  record  in  the  mountainous 
category  (T5)  leaving  eight  records  in  the  data  set.  Figure  25  shows  the  selection  of 
topography  categories. 
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Figure  25:  The  Topography  Form.  The  Topography  Form  allows  the  user  to  filter 
the  data  set  based  on  the  predominant  topography  of  the  OOU.  This  figure  shows  a 
minimum  topography  of  T1  and  a  maximum  topography  of  T4  selected. 

f.  Filter  by  Soil  Type 

The  next  form  is  the  Soil  Type  Form  (Figure  26).  Soil  categories  have 
ORC A/RACER  difficulty  factors  (Table  10)  and  the  cost-risk  models  labels  these 
categories  as  SI  through  S6. 
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Soil  Type 

Difficulty  Factor 

SI:  Sand 

0 

S2:  Gravel  Sand  mixture 

0.2 

S3:  Silt  or  Sand-Silt 

0.3 

S4:  Clay/Sand  or  Clay  Silt 

0.5 

S5:  Clay/Sand  with  stone 

0.6 

S6:  Clay 

0.7 

S7:  Rock 

0.8 

Table  10:  Soil  Categories  and  Associated  Difficulty  Factors.  In  accordance  with  the 
RACER/ORCA  model,  the  cost-risk  model  ranks  soil  types  by  difficulty  factors. 

The  scenario  site  is  in  the  S2  to  S3  categories,  and  since  the  user  does  not  want  to 

eliminate  too  many  records  he  selects  S 1  through  S5  to  eliminate  the  very  difficult 

cleanups  in  the  clay  and  rock  categories.  Using  these  criteria  the  filtering  removes  no 

further  records. 
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Query  Records  Based  On  Soil  Type 


a— 

1  Soil  is  ranked  by  increasing  difficulty  in  the  following  categories. 

SI 

Sand 

S2 

Gravel  Sand  mixture 

S3 

Silt  or  Sand-Silt 

S4 

Qay/Sand  or  Gay  Silt 

S5 

Gay/Sand  with  stone 

S6 

Gay 

S7 

Rock 
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Figure  26:  The  Soil  Type  Form.  The  Soil  Type  Form  allows  the  user  to  filter  the 
data  set  based  on  the  OOU’s  predominate  soil  type.  This  figure  shows  a  minimum 
soil  type  of  SI  and  a  maximum  soil  type  of  S4. 

g.  Filter  by  Ordnance  Type 

The  final  filtering  category  is  ordnance.  Here  again,  difficulty  factors  for 
ordnance  types  exist  as  specified  in  RACER/ORCA.  Table  1 1  lists  ordnance  types 
ranked  by  difficulty  factor. 
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Ordnance  Category 

Difficulty  Factor 

01:  Bombs  (Practice) 

0.1 

01:  Grenades  (Practice) 

0.1 

01:  Landmines  ( Practice) 

■■HEflIIIIHHi 

02:  Detonators,  Blasting  Caps,  Fuzes,  Boosters,  Bursters 

03:  Grenades,  Hand  and  Rifle  ( Explosive) 

0.4 

04:  Small  Arms 

0.5 

05:  Landmines  (Explosive) 

0.7 

06:  Rockets,  Guided  Missiles  ( Explosive) 

0.8 

07:  Medium/Large  Caliber  Rounds  (Explosive) 

1.0 

07:  Bombs  (Explosive) 

1.0 

Table  11:  Ordnance  Categories  and  Associated  Difficulty  Factors.  The  cost>risk 
model  ranks  ordnance  RACER/ORCA  difficulty  factors. 

Often  there  is  uncertainty  in  the  amount  and  type  of  ordnance  that  is  present  in  an  OOU. 
In  some  cases  a  range  contains  only  one  specific  type  of  ordnance,  at  other  times  the 
contaminated  area  has  everything  from  practice  rounds  to  high  explosive  submunitions  to 
2000-pound  air  delivered  bombs.  In  order  to  filter  OOUs  by  ordnance  type,  each  OOU 
in  the  database  has  a  mean  ordnance  difficulty  factor.  Equation  5.1  calculates  the 
ordnance  difficulty  factor  by  a  weighted  average  of  the  proportion  of  an  ordnance  type 
and  its  difficulty  factor: 

Expected  Proportion  *  Difficulty  Factor  (5.1) 

OT 

where  OT  is  the  ordnance  type 

In  many  cases  it  is  difficult  for  the  user  to  estimate  the  type  of  ordnance 
and  proportions  of  each  type  of  ordnance;  however,  there  is  usually  enough  information 
to  rule  out  extreme  records.  In  the  scenario  the  ordnance  difficulty  factor  has  a  value 
equal  to  a  minimum  of  (0.4).  Ruling  out  records  below  (0.3)  is  a  viable  solution.  Figure 
27  shows  the  Ordnance  Form. 
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dtdnancse  Difficufty  Foitre  . 


Figure  27:  The  Ordnance  Form.  The  Ordnance  Form  allows  the  user  to  filter  a 
range  of  ordnance  types  based  on  their  difficulty  factor.  The  figure  shows  the 
minimum  factor  set  at  0.4  and  the  maximum  factor  set  at  1. 

To  help  decide  the  ordnance  difficulty  factor  for  the  contaminated  area, 
the  user  can  click  the  Calculate  ODF  Button  in  the  center  of  the  form.  This  button 
launches  the  “Ordnance  Difficulty  Calculator”  to  determine  the  mean  difficulty  factor 
using  the  inputs  and  equation  5.1.  Figure  28  is  the  Ordnance  Difficulty  Calculator. 
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Figure  28:  The  Ordnance  Difficulty  Calculator  Form.  To  help  the  user  determine 
the  average  ordnance  difficulty  factor,  the  user  checks  the  types  of  ordnance  and 
fills  in  the  percentages  of  that  type.  The  user  assumes  that  the  area  contains  34% 
03, 33%  06,  and  33%  07.  The  bottom  value  (0.73)  is  the  mean  difficulty  factor. 

Once  the  user  is  satisfied  with  the  filter  criteria  the  values  can  be 
reviewed  by  selecting  the  Values  Form. 
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Figure  29:  The  Values  Form.  The  user  can  view  the  Values  Form  at  anytime  to  see 
the  filtering  criteria. 

2.  Step  Two:  Historical  Analysis 

With  the  data  set  reduced  down  to  a  satisfactory  group  of  similar  cleanups,  the 
user  selects  “Historical  Analysis”  from  the  “Cost_Analysis”  pull-down  menu.  This 
selection  generates  three  additional  sheets  in  the  workbook.  These  sheets  are 
“DescStats,”  “Histograms,”  and  “Correlation.” 

a.  Descriptive  Statistics  of  Similar  Cleanups 

The  “DescStats”  sheet  is  the  descriptive  statistics  on  each  cost  element  of 
the  filtered  set.  The  cost-risk  model  presents  the  minimum,  median,  average,  maximum. 
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and  standard  deviation  of  the  data  set  in  a  matrix.  Table  12  is  the  output  of  “DescStats” 
for  the  scenario  cleanup. 


Per  Acre  Costs 

Minimum 

Median 

Mean 

Max 

Stan  Dev 

EECA  Labor  Cost 

$650 

$900 

$999 

$1,340 

$237 

EECA  Material  Cost 

$60 

$200 

$354 

$1,070 

$348 

EECA  Travel/Per  Diem  Cost 

$250 

$380 

$526 

$1,530 

$450 

RA  Labor  Cost 

$3,170 

$4,440 

$4,907 

$6,700 

RA  Material  Cost 

$1,720 

$5,240 

$1,700 

RA  Travel/Per  Diem  Cost 

$1,210 

$1,840 

$2,571 

$7,520 

$2,218 

Table  12:  Output  of  Historical  Analysis  “DescStats”  Worksheet.  This  table . 
appears  in  the  worksheet  “DescStats”  after  running  a  historical  analysis  on  the 
filtered  set  of  similar  cleanups.  The  worksheet  contains  descriptive  statistics  of  the 
individual  cost  elements. 

b.  Histograms 

The  second  sheet  contains  the  Histograms  for  each  of  the  six  cost 
elements.  The  histograms  give  a  graphical  representation  of  what  has  been  spent  in  a 
specific  area  for  similar  cleanups.  Figure  30  is  a  view  of  this  worksheet. 
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Figure  30:  Output  of  Historical  Analysis  Histograms.  This  figure  shows  the  second 
sheet  that  the  model  generates  when  performing  a  historical  analysis.  Visible  in  the 
figure  are  two  out  of  the  six  cost  element  histograms.  The  user  can  use  this  visual 
information  to  get  a  feel  for  the  empirical  distribution  and  make  inferences  on  the 
true  distribution 

c.  Correlation  Matrix 

The  final  sheet  produced  during  the  historic  data  analysis  is 
“Correlations.”  This  sheet  contains  the  correlation  matrix  from  the  filtered  data  set.  The 
matrix  shows  how  each  cost  element  correlates  with  every  other  cost  element.  The 
correlation  matrix  for  the  unfiltered  data  set  and  the  scenario  data  set  appear  in  Table  13. 
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All  Records 

EECA  Labor 

EECA 

Material 

EECA 

Travel 

PerDiem 

RA  Labor 

RA  Material 

RA  Travel 
PerDiem 

EECA  Labor 

1.000 

EECA 

Material 

0.710 

1.000 

EECA 

Travel 

PerDiem 

0.798 

0.907 

1.000 

RA  Labor 

1 

mf 

i 

0.710 

0.799 

1.000 

RA  Material 

0.714 

1 

■ 

0.908 

0.714 

1.000 

RA  Travel 
PerDiem 

0.799 

0.906 

0.800 

0.907 

1.000 

Filtered  Set 

EECA  Labor 

EECA 

Material 

EECA 

Travel 

PerDiem 

RA  Labor 

RA  Material 

RA  Travel 
PerDiem 

EECA  Labor 

1.000 

EECA 

Material 

0.077 

1.000 

EECA 

Travel 

PerDiem 

0.253 

1.000 

RA  Labor 

■ 

■ 

0.075 

0.258 

1.000 

RA  Material 

0.082 

— i 

0.938 

0.080 

1.000 

RA  Travel 
PerDiem 

0.257 

0.263 

0.937 

1.000 

Table  13:  Correlation  Matrices  for  all  Records  and  the  Scenario  Records.  The 
upper  matrix  is  the  correlation  matrix  that  the  model  generates  when  performing  an 
historical  analysis  on  all  the  records  in  the  database.  The  Lower  matrix  is  the 
correlation  matrix  for  the  scenario  data  set.  The  off  diagonal  correlation 
coefficients  that  equal  1.0  (highlighted  values)  result  from  how  the  EECA  data  was 
generated. 

The  values  highlighted  in  the  table  have  a  correlation  value  of  1.0  because  taking  2%  of 
the  corresponding  RA  values  generates  the  unavailable  EECA  data.  One  should  note  the 
high  correlation  in  the  unfiltered  set  for  RA  data.  Material  and  Labor  have  a  value  of 
0.71,  Labor  and  Travel/per  Diem  have  a  value  of  0.8,  and  Material  and  Travel/per  Diem 
have  a  value  of  0.90.  For  the  filtered  set  these  values  were,  0.08, 0.263,  and  .937 


respectively.  The  coefficients  in  the  filtered  set  tend  to  be  much  smaller  except  for 


Material-Travel/per  Diem.  The  small  correlation  in  the  filtered  set  is'likely  a  result  of  a 
few  outliers  in  the  small  set.  This  thesis  recommends  that  the  user  input  correlation 
coefficients  closer  to  the  unfiltered  set.  As  data  sets  of  similar  cleanups  become  larger 
the  user  can  choose  the  correlation  coefficients  from  the  filtered  data  set. 

3.  Step  Three:  Create  Cost  Element  Probability  Distributions 

a.  Choosing  Alpha 

The  next  step  prior  to  simulation  is  to  create  weighted  probability  density 
functions  for  each  cost  element  (the  cost-risk  model  uses  a  linear  combination  of  the 
empirical  distribution  and  the  user’s  supplied  triangular  distribution  to  form  a  weighted- 
distribution).  The  user  must  determine  the  weighting  factor  or  for  each  cost  element 
distribution  (see  Equation  4.1).  The  weighting  factor  ^expresses  the  user’s  faith  in  the 
triangular  distribution.  For  example  when  oris  equal  to  n  (the  number  of  records)  the 
empirical  distribution  is  equal  in  weight  to  the  triangular  distribution.  When  cr  equals 
zero  the  formula  gives  no  weight  to  the  triangular  distribution  and  when  oris  very  large 
the  weight  shifts  to  the  triangular  distribution.  Figure  3 1  visually  shows  the  effects  of 
varying  a.  The  first  graph  Figure  3 1  is  the  histogram  of  the  weighted  PDF  with  alpha 
equal  to  zero.  The  second  histogram  is  with  an  alpha  equal  to  the  number  of  records  (19). 
The  final  histogram  uses  an  alpha  of  100. 
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K&stogram 


Figure  31:  Increasing  Alpha  and  the  Shape  of  the  Weighted  PDF.  This  example  uses 
the  same  empirical  distribution  and  triangular  distribution  but  varies  the  alpha. 
With  a  =  0  the  first  histogram  mirrors  the  empirical  PD,F.  With  an  a  =  n  the  middle 
PDF  is  slightly  triangular  with  spikes  at  the  discrete  values.  At  a  =  100  the  last  PDF 
is  nearly  triangular  with  some  slight  spikes. 

b.  Inputting  Triangle  Parameters 

By  selecting  “Create  Cost  Distributions”  from  the  “Cost_Analysis”  menu, 
the  cost-risk  model  presents  the  user  with  the  “Set  Triangular  PDFs”  form.  Visible  on 
this  form  are  the  statistics  for  the  cost  element  (minimum,  mean,  average,  maximum), 
and  input  boxes  to  set  alpha  and  the  triangular  distribution’s  parameters  (minimum  value, 
most-likely  value,  and  maximum  values).  The  model  sets  the  default  value  for  or  equal 
to  the  number  of  records,  it  sets  the  triangle’s  minimum  and  maximum  to  80%  of  the 
empirical  distribution’s  minimum  and  120%  of  the  empirical  distribution’s  maximum, 
and  sets  the  triangle’s  most-likely  value  equal  to  the  empirical  distribution’s  mean  value. 
The  default  values  are  only  starting  points  and  may  not  be  realistic  or  desirable.  The  user 
should  carefully  consider  each  cost  element  when  setting  or  and  the  triangular 
distribution’s  parameters. 
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Figure  32:  The  Set  Triangle  PDFs  Form.  The  user  uses  this  form  to  set  o;  and  set 
the  triangular  distribution’s  parameters  (minimum  value,  most-likely  value,  and 
maximum  value).  The  form  shows  the  statistics  for  each  cost  element  and  the  user 
can  click  on  the  Show  Histogram  Button  to  view  the  histogram  of  the  Altered  data. 

The  form  displays  the  name  of  the  cost  element  in  the  center,  and  the  user  uses  the  scroll 

bar  next  to  the  cost  element’s  name  to  cycle  through  all  of  the  cost  elements.  For  the 

scenario,  the  form  shows  that  seven  similar  cleanup  records  are  found.  For  the  Removal 

Action  labor  costs  the  user  may  feel  that  the  hypothesized  triangular  distribution  is  as 

valuable  as  the  empirical  distribution,  and  sets  alpha  equal  to  seven  (the  number  of 

records).  An  alpha  equal  to  the  number  of  records  gives  equal  weighting  to  the  empirical 

distribution  and  the  traingular  distribution.  The  user  looks  at  the  statistics  along  the  left 

side  of  the  form  and  then  reviews  the  histogram  for  the  cost  element  (Figure  33). 
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Figure  33:  The  Histogram  for  Removal  Action  Labor  Costs  in  Dollars/Acre.  The 
histogram  for  RA  labor  costs  does  not  reveal  much  of  a  trend  in  the  data,  but 
provides  a  visual  depiction  of  costs  of  similar  cleanups. 


Assume  that  a  cost  analyst  believes  that  contractors  never  charge  less  than  $2,500/acre 


and  that  the  government  never  enters  into  contracts  for  this  type  of  cleanup  for  more  than 


$8,0C)0/acre.  To  provide  more  insight  the  user  runs  the  RACER/ORCA  model  and  gets  a 
point  estimate  of  $6,200/acre  for  the  labor  costs  .  The  analyst  builds  a  triangular 
distribution  based  on  the  RACER/ORCA  estimate  and  the  historic  data.  The  user  sets  the 
minimum  of  the  triangular  distribution  at  $2, 500/acre,  the  center  at  $5, 000/acre,  and  the 
maximum  at  $8, 000/acre.  The  user  then  enters  these  values  at  the  bottom  of  the  form. 
The  same  methodology  applies  for  all  cost  elements.  When  the  user  gets  to  EE/CA 
material  costs  he  feels  that  the  data  accurately  reflects  the  true  distribution  (unlikely  with 
only  seven  records)  and  decides  to  put  all  of  the  weight  onto  the  empirical  data  by  setting 


a  for  this  cost  element  to  zero.  Conversely,  policy  changes  have  altered  how  per  Diem  is 


paid  and  the  user  determines  that  the  empirical  data  is  too  high  and  not  reflective  of 
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current  costs.  The  analyst  thus  sets  orfor  “RA  Travel/Per  Diem”  cost  to  100,  heavily 
weighting  the  user  supplied  triangular  distribution. 

When  the  user  clicks  the  Done  button  the  model  creates  the  weighted 
PDFs  for  each  cost  element  composed  of  10,000  random  points  from  the  empirical  and 
triangular  distribution.  The  cost  element’s  a  determines  how  many  points  come  from 
each  distribution.  If  the  weighting  is  4/5  for  the  empirical  and  1/5  for  the  triangular 
distribution  then  the  model  generates  8,000  random  points  from  the  empirical  distribution 
and  2,000  points  from  the  triangular  distribution.  Code  is  written  in  visual  basic  using 
the  inverse  transform  method  to  create  these  distributions.  Any  simulation  package  can 
then  draw  with  equal  probability  from  these  10,000  numbers  to  form  the  weighted- 
distribution. 

4.  Step  Four:  Run  Simulation 
a.  “OutPutReport*’ 

The  cost-risk  model  creates  two  additional  sheets  at  this  time: 
“OutPutReport”  and  “Siminputs.”  The  “OutPutReport”  sheet  summarizes  the  statistical 
data  for  each  cost  element,  the  triangular  distributions  for  each  cost  element,  the  filtering 
criteria,  and  the  correlation  matrix.  Figure  34  shows  the  information  that  the 
“OutPutReport”  sheet  contains. 
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Figure  34:  The  “OutPutReport”  Worksheet.  The  “OutPutReport”  worksheet 
shows  the  statistics  for  each  cost  element,  the  triangular  distribution  parameters,  a 
for  each  cost  element,  the  filter  criteria,  and  the  correlation  matrix  for  the  found  set. 
The  correlation  matrix  is  not  visible  in  the  figure. 


h.  **SimInputs” 

The  worksheet  “Simlnputs”  (See  Figure  35)  contains  formulas  composed 
of  the  random  variables,  and  the  simulation  software  uses  this  sheet  to  run  the  simulation. 
The  worksheet’s  formulation  works  with  @Risk  but  can  be  easily  modified  to  interface 
with  other  simulation  packages. 
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Figure  35:  The  “Simlnputs”  Worksheet.  The  “Siminputs”  worksheet  sets  up  the 
simulation.  Cells  D3  to  D8  draw  random  samples  from  the  weighted  PDFs  of  the 
corresponding  cost  elements.  Cell  Dll  is  the  sum  of  all  RA  only  cost  elements,  and 
D12  is  the  sum  of  all  cost  elements. 

Although  different  simulation  packages  require  slightly  different 
formulation,  in  general  the  inputs  are  the  cost  element  PDFs  and  the  outputs  are  functions 
of  these  inputs.  A  great  advantage  of  simulation  is  that  multiple  functions  of  the  inputs 
can  be  created.  The  “Siminput”  sheet  provides  an  example  of  this  with  one  output  being 
the  sum  of  all  cost  elements  and  one  being  the  sum  of  only  the  RA  inputs.  In  this  way 
one  derives  an  RA  total  cost  per  acre  PDF  and  a  combined  RA/EECA  total  cost  per  acre 
PDF  (both  PDFs  are  in  dollars/acre).  Further  the  user  could  write  another  function  that 
sums  up  the  inputs  and  then  multiplies  them  by  the  acreage  of  the  contaminated  area. 

This  final  formulation  becomes  the  total  cost  PDF  in  dollars.  After  the  user  writes  any 
additional  output  formulas  he  desires,  he  enter  the  inputs  and  outputs  into  @Risk  along 
with  the  correlation  matrix  and  then  runs  the  simulation. 

F.  JUSTIFICATION  OF  FILTERING 

To  demonstrate  that  the  filtering  can  accurately  locate  cleanups  with  similar  costs, 
the  thesis  mns  two  additional  scenarios.  The  thesis  considers  the  original  scenario  as  a 
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medium  difficulty  cleanup  because  it  contains  a  mixture  of  difficult  ^d  easy  OOU 
characteristics.  The  density  is  low  to  medium;  the  vegetation  is  high  in  difficulty;  the 
depth  of  clearance  and  ordnance  type  is  medium  in  difficulty;  and  the  terrain  and  soil 
t5^es  are  easy.  The  “hard”  and  “easy”  scenarios  contain  the  same  amount  of  records  (7) 
as  the  original,  “medium,”  scenario.  The  “hard”  scenario  filters  the  records  by  setting  all 
the  OOU  characteristics  in  the  difficult  ranges  (i.e.,  high  density,  high  vegetation 
difficulty  factor,  deep  depth  of  clearance,  etc.).  The  “easy”  scenario  does  the  converse, 
setting  the  OOU  characteristics  to  the  lower  degrees  of  difficulty.  For  all  cost  elements 
and  for  all  summary  statistics,  the  “easy”  cleanup  has  lower  values  than  the  “medium” 
cleanup  and  the  “medium”  cleanup  has  lower  values  than  the  “difficult”  cleanup.  Figure 
36  shows  the  minimum,  maximum,  average,  and  median  value  for  each  cost  element  of 
each  scenario. 
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Figure  36:  Summary  Statistics  for  “Easy,”  “Medium,”  and  “Hard,”  Scenario.  The 
model  develops  three  scenarios:  “Easy”  cleanup,  “Medium”  cleanup  and  “Hard” 
cleanup.  The  Altering  process  Ands  similar  cleanups  as  can  be  seen  in  the  summary 
statistics.  For  each  cost  element  the  minimum  cost,  the  maximum  cost,  the  median 
cost,  and  average  cost  increase  as  difAculty  increases. 
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VI.  SIMULATION  OUTPUT  AND  INTERPRETATION 


A.  SINGLE  CLEANUP  RESULTS 

The  Monte  Carlo  simulation  is  run  for  the  Chapter  V  scenario  cleanup  using  the 
“all  records”  correlation  matrix  at  the  top  of  Table  18,  and  by  setting  @Risk  for  10,000 
iterations.  To  emphasize  the  necessity  of  considering  correlation  effects,  the  thesis  shows 
the  results  for  an  identical  simulation  that  assumes  zero  correlation  among  cost  elements. 
Appendix  C  contains  the  output  report  from  the  scenario.  Figure  37  shows  how  Monte 
Carlo  sampling  forms  a  total  cost  PDF  from  the  cost  element  PDFs. 
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The  @Risk  software  package  generates  the  total  cost  PDF  shown  at  the  bottom  of  the 
previous  figure.  Since  only  a  few  cost  elements  exist  in  the  simulation,  the  total  cost  PDF 
does  not  approach  the  normal  distribution  shape,  but  the  distribution  does  begin  to 
smooth  out. 

To  portray  the  importance  of  considering  correlation.  Figure  38  compares  the 
uncorrelated  (left  side)  and  correlated  (right  side)  total  cost  probability  density  functions. 
As  is  expected,  the  PDF  for  the  uncorrelated  case  is  more  symmetrical  and  less  spread  out 
(less  variance). 


Removal  Action/EECA  Total  Cost 


Uncoirelated 


0.154 

0.123,. 

0.092 

0.061 

0.031 


0.000. 


4.52  7.48  10.44  13>t0  1636  19.32  22.28 
Values  inThousands 


Removal  Action/EECA  Total  Cost  i 
Correlated 

b.123^ 

0^098 
0.074 
a049 
0.025 


0.000. 


4.24  7.81  14.97  18.M  22.12  25.69  ; 

Values  in  Thousands  , ;  .  ' 


Figure  38:  Comparison  of  Uncorrelated  and  Correlated  Total  Cost  Per  Acre  PDFs. 
The  comparison  of  the  uncorrelated  (left  side)  and  correlated  (right  side)  simulation 
outputs  shows  that  uncorrelated  random  variables  produce  a  tighter  distribution. 
Costs  are  in  dollars/acre. 

The  comparison  shows  that  by  ignoring  correlation,  the  analyst  assumes  an  artificial 
“reduction  in  risk.”  Reduction  in  risk  refers  to  the  fact  that  there  is  a  reduced  chance  of 
getting  values  in  the  tail 

In  addition  to  the  PDF,  the  simulation  software  generates  the  summary  statistics 
for  the  total  cost  distribution.  Table  14  lists  the  summary  statistics  for  the  correlated  and 
uncorrelated  scenario  cleanup. 
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Statistic 

Correlated  Value 

Uncorrelated  Value 

Minimum 

$4,239 

$4,518 

Maximum 

$25,694 

$22,284 

Mean 

$11,715 

$11,711 

Std  Deviation 

$4,260 

$2,415 

Mode 

$8,400 

$9,600 

Table  14:  Correlated  and  Uncorrelated  Summary  Statistics  for  the  Scenario 
Cleanup.  The  simulation  software  produces  summary  statistics  for  all  input  and 
output  cells.  This  table  lists  the  summary  statistics  for  the  total  cost  per  acre  of 
performing  an  EECA  and  RA  on  the  contaminated  area. 

The  summary  statistics  for  the  correlated  inputs  show  that  for  the  scenario  cleanup  the 
simulation  estimates  total  cost  per  acre  to  fall  between  $4,239/acre  and  $25,694/acre. 

The  results  estimate  the  most-likely  value  to  be  around  $8,400/acre  (mode),  but  shows  in 
the  long  run  an  average  around  $1 1,715/acre  (mean).  As  is  expected,  both  the  correlated 
and  uncorrelated  scenarios  have  the  same  mean  but  the  standard  deviation  is  much 
greater  for  the  correlated  case. 

Before  the  cost  analyst  can  make  informed  decisions  about  costs,  he  needs  the 
cumulative  probability  distribution  for  the  total  cost.  The  cumulative  probability 
distribution  is  also  an  output  of  the  simulation  software,  and  Table  15  shows  tabulated 
values  of  the  cumulative  probability  distribution. 
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X 

(Correlated) 

X 

(Uncorrelated) 

Probability  that  True 
Total  Cost  is  Less  than  X 

$6,141 

$8,272 

0.05 

$6,881 

$8,853 

0.10 

$7,575 

$9,256 

0.15 

$8,061 

$9,623 

0.20 

$9,959 

0.25 

$8,910 

$10,250 

0.30 

$9,343 

$10,544 

0.35 

$9,803 

$10,804 

0.40 

$10,327 

$11,114 

0.45 

$10,794 

$11,433 

0.50 

$11,340 

$11,745 

0.55 

$11,924 

$12,067 

0.60 

$12,586 

$12,432 

0.65 

$13,355 

$12,789 

0.70 

$14,221 

$13,219 

0.75 

$15,219 

$13,692 

0.80 

$16,527 

$14,284 

0.85 

$18,172 

$15,008 

0.90 

$20,005 

$16,077 

0.95 

$25,694 

$22,284 

1.00 

Table  15:  The  Total  Cost  Per  Acre  Cumulative  Distribution  for  the  Chapter  V 
Scenario.  The  simulation  software  generates  the  table  of  the  cumulative 
distribution.  The  value  in  the  far  right  column  gives  the  probability  that  the  total 
cost  per  acre  is  less  than  the  corresponding  value  in  the  left  column.  To  emphasize 
the  effects  of  correlated  random  variables,  the  table  gives  values  for  both  the 
correlated  simulation  and  the  uncorrelated  simulation. 

The  cumulative  distribution  shows  the  probability  that  the  cost  of  cleanup  will  fall  below 
a  given  value.  As  mentioned  previously,  there  is  an  artificial  reduction  in  the  risk  when 
not  considering  correlation.  For  example  the  cost  analyst  may  be  asked,  “what  is  the 
probability  of  costs  exceeding  $15,000  per  acre  ($675,000  total  for  45  acres)?”  Using 
the  false  assumption  of  no  correlation  of  cost  elements,  the  analyst’s  answer  is  about 
10%.  The  proper  correlated  result  shows  an  answer  closer  to  20%. 
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With  the  above  information  the  cost  analyst  can  make  informed  decisions  about 
the  estimated  cost  and  the  risks  associated  with  it.  For  example,  the  analyst  may  feel  that 
that  $8, 600/acre  (mode)  is  the  most  likely  cost  for  the  EE/CA  and  RA,  but  he  is  also 
aware  that  the  probability  of  exceeding  this  cost  is  roughly  75%.  For  budgeting  purposes 
the  analyst  may  choose  a  value  closer  to  $1 1,000/acre  (median),  because  the  probability 
is  equally  likely  that  the  cost  is  greater  than  or  less  than  this  value. 

B.  MULTIPLE  CLEANUPS 

Although  users  of  the  cost-risk  model  need  estimates  for  individual  OOUs,  they 
more  often  desire  project  level  estimates.  Since  projects  consist  of  one  or  more  OOUs, 
the  user  can  use  the  individual  OOU  estimates  to  produce  project  estimates.  In  an 
individual  OOU  estimate  each  iteration  of  the  Monte  Carlo  simulation  forms  an  input  for 
the  OOUs  total  cost  probability  distribution,  and  the  model  maintains  these  values  after 
the  simulation.  For  example,  setting  the  simulation  to  10,000  iterations  forms  10,000 
random  observations  from  the  OOUs  total  cost  distribution.  The  simulation  software  can  . 
sum  the  individual  total  cost  probability  distributions  for  several  OOUs  to  form  a  single 
estimate  for  the  combined  cleanup.  To  demonstrate  this  capability,  consider  a  cleanup 
project  consisting  of  three  OOUs:  the  “hard,”  “medium,”  and  “easy”  scenario  cleanups 
presented  in  Section  V.  The  “hard”  cleanup  is  10  acres  in  size,  the  “medium”  is  45  acres, 
and  the  “easy”  cleanup  is  60  acres.  To  estimate  the  entire  project  cost,  the  analyst  first 
produces  estimates  for  each  individual  OOU.  Next,  Monte  Carlo  simulation  sums  the 
three  OOU  total  cost  distributions.  Each  iteration  of  the  simulation  consists  of  randomly 
drawing  a  value  from  each  of  the  three  total  cost  distributions  and  sununing  them 
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together.  The  analyst  converts  the  units  of  each  input  distribution  from  “dollars  per  acre” 
to  “dollars”  so  that  the  resulting  distribution  of  the  simulation  is  the  total  “dollar  cost”  of 
cleaning  the  three  OOUs.  Figure  39  visually  depicts  the  simulation. 


Removal  Action/EECA  Total  Cost 
60  Acre  Easy  Cleanup 

0.»05 


204.#-  -  287»»  ; '  5570.6 ;  ^  si  453.5 ;  '  536.4 

Values  In  Thousaixfs 


Estimate  for  Three  OOU  Cleanup 
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Figure  39:  A  “Multi-OOU”  Estimate.  Users  most  often  desire  a  combined  estimate 
for  several  cleanups.  The  user  initially  produces  individual  OOU  distributions  and 
then  uses  the  OOU  distributions  as  the  inputs  for  the  “multi-OOU’’  estimate.  This 
figure  shows  a  project  that  consists  of  three  OOUs:  60-acre  “easy”  cleanup,  45-acre 
“medium”  cleanup,  and  10-acre  “hard”  cleanup. 

VII.  The  “three-OOU”  distribution  shows  that  the  most  likely  cost  for  cleanup  of 
the  three  OOUs  is  roughly  $1.7  million.  Appendix  D  contains  the  simulation 
summary  for  the  “three-OOU”  estimate.  The  summary  includes  the  cumulative 
distribtuion  and  the  summary  statistics. 


83 


84 


RECOMMENDATIONS  AND  CONCLUSIONS 


This  thesis  demonstrates  a  model  for  conducting  cost-risk  analysis  of  UXO 
cleanups.  This  research  shows  that  UXO  conteiminated  areas  with  similar  characteristics 
have  similar  costs.  Given  appropriate  data  the  model  is  capable  of  finding  historic 
cleanups  that  have  characteristics  similar  to  a  proposed  cleanup.  With  the  historic 
records,  the  cost  analyst  has  descriptive  information  to  assist  him  in  building  cost  element 
PDFs.  Using  the  output  of  Monte  Carlo  simulation  of  cost  element  PDFs,  the  cost 
analyst  has  the  tools  to  make  a  cost  estimate  that  incorporates  cost-risk  analysis. 

A.  RECOMMENDATIONS 

1.  Model  Advantages  and  Disadvantages 

The  reader  should  consider  the  cost-risk  model  as  a  prototype.  The  main  shortfall 
of  the  model  is  the  limited  amount  of  historic  data  records  it  contains.  This  lack  of 
records  reduces  the  users  ability  to  find  a  satisfactory  pool  of  similar  cleanups.  The 
USAESCH  is  implementing  procedures  that  will  make  historic  cost  data  more  available 
and  thus  allow  for  the  population  of  a  cost  database.  The  cost-risk  model  requires  some 
modification  to  the  OED  in  order  to  obtain  the  information  in  the  proper  format. 

Appendix  A  outlines  required  OED  modifications. 

The  second  shortfall  of  the  cost-risk  model  is  in  the  cost  elements  that  the  model 
uses.  These  cost  elements  reflect  the  cost  information  format  that  is  available,  and  they 
do  not  represent  an  ideal  breakdown  of  costs.  The  cost  elements  have  high  correlation, 
and  because  of  their  broad  nature  may  be  difficult  for  an  analyst  to  estimate.  The 
advantage  of  a  large  numbe  of  cost  elements  is  that  smaller  “pieces”  are  usually  more 
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intuitive  for  an  analyst  to  estimate.  For  example  “Labor”  costs  may  be  more  difficult  to 
estimate  than  the  components:  “Supervisor  Labor,”  “Ordnance  Technician  Labor,” 
“Engineering  Labor,”  and  “Quality  Control  Technician  Labor”  costs.  Appendix  E. 
contains  the  72-line  work  breakdown  structure  (WBS)  that  is  the  basis  for  reporting  costs 
in  the  OED.  This  thesis  recommends  that  future  cost  models  or  modifications  to  this 
model  attempt  to  use  cost  elements  that  are  a  compromise  between  the  cost-risk  model 
cost  elements  and  the  WBS  costs.  Regardless  of  the  cost  reporting  fidelity,  the 
USAESCH  should  establish  a  consistent  digital  cost  reporting  format  that  can  be  entered 
into  a  database.  Currently,  contractors  submit  most  cost  information  in  “hard-copy” 
reports.  Cost  analysts  must  enter  this  information  manually  into  the  OED.  The  cost-risk 
model  design  allows  for  easy  modification  to  accept  additional  cost  elements. 

One  advantage  of  the  model  is  that  it  refines  itself  as  more  data  becomes 
available.  Experts  in  this  field  believe  that  as  detection  technology  improves,  ordnance 
removal  costs  will  drastically  decrease.  Future  records  added  to  the  database  would 
reflect  these  lower  costs.  Conversely,  a  parametric  cost  relationship  built  today  may 
become  obsolete  due  to  these  cost  changes.  More  records  also  mean  that  the  set  of 
similar  cleanups  will  be  larger. 

Another  advantage  of  the  model  is  that  its  design  allows  for  records  to  be  updated 
from  a  text  file.  Maintainers  of  ordnance  information  could  download  the  text  file  from 
the  OED  or  another  ordnance  management  system  and  then  disseminate  it  to  users  around 
the  country.  The  Defense  Environmental  Network  and  Information  Exchange  (DENIX) 
is  the  ideal  means  for  disseminating  this  data.  This  internet  service  provides  DoD 
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personnel  access  to  environmental  legislative,  compliance,  restoration,  cleanup,  and  DoD 
guidance  information.  It  serves  as  a  central  electronic  "meeting  place"  where  information 
can  be  exchanged  among  environmental  professionals  worldwide.  Users  of  the  cost-risk 
model  could  easily  retrieve  updated  data  from  this  exchange  while  the  DOD  could  still 
control  who  is  permitted  access  to  the  data. 

Finally,  the  model’s  “Historical  Analysis”  function  provides  the  USAESCH  a 
means  to  analyze  what  was  done  in  the  past.  Even  if  the  user  is  not  performing  a  cost 
estimate,  this  feature  can  be  used  to  answer  many  questions.  For  example  the  user  could 
manually  select  records  from  a  specific  contractor  to  analyze  whether  or  not  their 
“Labor”  costs  differ  significantly  from  other  contractors. 

2.  Further  Research 

This  thesis  gives  insight  to  many  other  possible  areas  of  research.  Primarily, 
information  management  research  needs  to  be  done  to  determine  how  cost  information 
can  be  easily  acquired  and  managed.  The  OED  is  intended  to  gather  cost  information, 
but  it  appears  that  little  structure  was  developed  to  make  this  possible.  Further,  raw  cost 
information  provides  little  information  without  a  means  to  analyze  it.  The  cost-risk 
model’s  “Historical  Analysis”  function  provides  some  analysis,  but  these  features  can  be 
greatly  increased. 

Validation  of  this  model  is  also  needed.  USAESCH  is  currently  trying  to  validate 
the  RACER/ORCA  models  so  that  USAECH  can  delegate  RACER/ORCA’s  use  down  to 
the  individual  Regions  of  the  Army  Corps  of  Engineers.  This  task  proves  difficult 


87 


because  little  information  has  been  kept  on  initial  government  estimates  and  actual  costs 
[Young  1999]. 

Finally,  the  RACER/ORCA  models  could  be  easily  modified  to  perform  cost-risk 
analysis.  According  to  an  USAESCH  cost  analyst,  “These  models  give  accurate  results 
as  long  as  the  correct  assumptions  are  made  [Young  1999].”  Unfortunately  the  models 
have  no  means  for  a  user  to  express  uncertainty  in  their  assumptions.  The 
RACER/ORCA  models  could  be  modified  so  that  all  inputs  containing  uncertainty  have  a 
probability  distribution.  For  example,  a  user  must  specify  the  density  of  the  ordnance, 
but  this  is  often  uncertain  even  after  sampling.  Instead  of  entering  the  ordnance  density 
as  a  single  value,  the  user  could  enter  a  probability  distribution  of  the  ordnance  density. 
The  final  result  would  be  a  “bottom-up”  engineering  model  that  benefits  from  cost-risk 
analysis.  These  changes  are  feasible  since  the  RACER/ORCA  models  and  computer 
code  are  property  of  the  DOD.  Modifying  RACER/ORCA  may  prove  easier  to 
implement  than  the  cost-risk  model  presented  in  this  thesis.  These  models  already 
contain  extensive  features  for  project  management  and  are  already  familiar  to  many  users 
who  deal  with  environmental  restoration. 

B.  CONCLUSIONS 

The  Range  Rule  requires  the  services  to  dedicate  a  significant  amount  of 
additional  resources  to  an  already  monumental  problem.  Only  a  few  agencies  such  as 
USAESCH  have  experience  in  this  area,  but  the  cost-risk  model  gives  many  possible 
users  the  ability  to  use  USAESCH  historic  data  to  budget  and  estimate  the  cost  of 
ordnance  remediations.  Estimating  the  costs  of  environmental  restoration  of  ordnance 
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contaminated  sites  is  a  difficult  process  containing  many  unknowns.  The  probabilistic 
cost  model  of  this  thesis  provides  the  means  to  manage  this  uncertainty  and  risk. 


89 


90 


APPENDIX  A.  ADDITIONS  AND  MODIFICATIONS  TO  THE  ORDNANCE 

AND  EXPLOSIVES  DATABASE 

This  appendix  makes  recommendations  for  additions  and  modifications  to  the 
data  fields  contained  in  the  Ordnance  and  Explosives  Database  (OED).  These 
recommendations  allow  for  the  cost-risk  model  proposed  in  this  thesis  to  use  the  OED 
structure.  The  OED  is  an  attempt  by  the  United  States  Army  Engineering  and  Support 
Center,  Huntsville  (US AESCH)  to  create  a  repository  of  information  on  unexploded 
ordnance  cleanups.  The  database  is  not  complete  and  is  not-sufficiently  populated  with 
existing  information.  The  database  may  likely  be  the  framework  for  a  future  unexploded 
ordnance  management  system,  and  therefore  the  thesis  attempts  to  use  the  current 
structure  as  much  as  possible.  Further  OED  developments  can  incorporate  these 
changes.  This  appendix  breaks  down  the  data  fields  into  two  groups:  OOU  descriptive 
characteristics,  and  cost  information. 

A.  OOU  DESCRIPTIVE  CHARACTERISTICS 
1.  Soil 

The  OED  only  allows  for  three  descriptive  characteristics  of  soil  type.  The  OED 
should  expand  the  allowable  inputs  in  order  to  reflect  the  structure  of  the  ORCA/RACER 
cost  model.  The  ORCATRACER  model  assigns  difficulty  factors  to  each  soil  type,  and 
the  cost-risk  model  adopts  theses  rankings.  Table  16  recommends  the  allowable  soil 
inputs  and  associated  difficulty  factors. 
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Soil 

Difficulty  Factor 

Sand 

0 

Gravel  Sand  mixture 

0.2 

Silt  or  Sand-Silt 

0.3 

Clay/Sand  or  Clay  Silt 

0.5 

Clay/Sand  with  stone 

0.6 

Clay 

0.7 

Rock 

0.8 

Table  16:  Recommended  Soil  Categories  and  Associated  Difficulty  Factors.  This 
thesis  recommends  that  the  OED  incorporate  the  above  soil  categroies  and  the 
associated  difficulty  factors.  This  cost-risk  model  uses  the  above  structure.  From 
[BTG  Incorporated  (Delta  Research  Division).  Remedial  Action  Cost  Engineering 
and  Requirements  (RACER)  System  3.2.a.  Computer  Software,  1998] 

2.  Vegetation 

The  OED  does  not  allow  enough  descriptive  categories  for  the  vegetation  of  an 
OOU.  Similar  to  soil  type,  the  current  OED  fields  do  not  provide  the  ability  to  capture 
the  difficulty  in  remediation  associated  with  differing  soil  types.  Again,  this  thesis 
recommends  the  adoption  of  the  RACER/ORCA  difficulty  factors.  Table  17  lists  the 
vegetation  categories  and  the  difficulty  factors. 


VEGETATION 

Difficulty  Factor 

Hat  barren  or  low  grass 

0 

Low  grass  and  few  shrubs 

0.1 

Heavy  grass  with  numerous  shrubs 

0.3 

Shrubs  with  some  trees 

0.3 

Heavy  shrubs  and  trees  or  forest 

0.5 

Table  17:  Recommended  Vegetation  Categories  and  Associated  Difficulty  Factors. 
The  thesis  recommends  that  the  OED  incorporate  the  above  vegetation  categories 
and  associated  difficulty  factors.  This  cost-risk  model  uses  the  above  structure. 
From  [BTG  Incorporated  (Delta  Research  Division).  Remedial  Action  Cost 
Engineerin£  and  Requirements  (RACER)  System  3.2.a.  Computer  Software  1998] 
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3.  Topography 

The  OED  only  allows  three  inputs  for  terrain:  “flat,”  “rolling,”  and  “rugged.”  In 
order  to  capture  the  difficulty  in  cleanup  due  to  topography  this  field  should  use  the  six 
topography  categories  used  in  RACER/ORCA.  Table  18  lists  these  categories  and  their 
associated  difficulty  factors. 


TOPOGRAPHY 

Difficulty  Factor 

Flat 

0 

Gently  Rolling 

0.1 

Heavy  Rolling 

0.3 

Flat  with  Gorges 

0.3 

Rolling  With  Gorges 

0.5 

Mountainous 

0.7 

Table  18:  Recommended  Topography  Categories  and  Associated  Difficulty  Factors. 
This  thesis  reconunends  that  the  OED  incorporate  the  topography  categories  and 
associated  difficulty  factors.  From  [BTG  Incorporated  (Delta  Research  Division). 
Remedial  Action  Cost  Engineerine  and  Requirements  (RACER)  System  3.2.a. 
Computer  Software  1998] 

4.  Density 

Density  usually  refers  to  the  amount  of  UXO  per  acre,  but  this  definition  may  not 
be  a  good  indicator  of  costs.  Technicians  must  remove  both  UXO  scrap  and  non-UXO 
scrap  from  a  contaminated  area,  and  false  anomalies  must  be  excavated  along  with  actual 
UXO.  The  relationship  between  UXO  density  and  non-UXO  scrap  density  and  its  effect 
on  costs  is  extremely  difficult  to  define.  The  cost-risk  model  records  density  in  total 
scrap  per  acre.  A  more  detailed  analysis  of  the  relationship  of  UXO  scrap  and  non-UXO 
scrap  and  its  effects  on  cost  is  beyond  the  scope  of  this  thesis  and  should  be  the  subject  of 
further  research. 
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5. 


Depth  of  Clearance 


The  OED  records  depth  at  which  individual  UXO  is  located.  There,  however,  is 
not  a  field  in  the  OED  that  captures  the  contracted  depth  of  clearance.  The  OED  should 
incorporate  a  depth  of  clearance  field. 

6.  Type  of  Ordnance 

The  OED  and  RACER/ORCA  use  the  same  inputs  for  the  type  of  ordnance  found 
in  the  OOU,  and  these  inputs  should  remain  unchanged.  The  cost-risk  model  uses  the 
RACER/ORCA  ordnance  difficulty  factors.  However,  since  many  ranges  are  multi-use,  a 
large  variety  of  ordnance  may  exist.  The  OED  should  represent  the  ordnance  difficulty 
by  a  weighted  average  of  each  type  of  ordnance  and  its  difficulty  factor.  Table  19  shows 
the  allowable  ordnance  inputs  and  their  difficulty  factors. 


TYPE_ORDNANCE 

Bombs,  Practice 

0.1 

Grenades,  Practice 

0.1 

Landmines,  Practice 

0.1 

Detonators,  Blasting  Caps,  Fuzes,  Boosters,  Bursters 

0.3 

Grenades,  Hand  and  Rifle,  Explosive 

0.4 

Small  Arms 

0.5 

Landmines,  Explosive 

0.7 

Rockets,  Guided  Missiles,  Explosive 

0.8 

Medium/Large  Caliber  (20mm  and  target) 

1.0 

Bombs 

1.0 

Table  19:  Recommended  Ordnance  Categories  and  Associated  Difficulty  Factors. 
The  OED  already  incorporates  the  above  ordnance  types  but  not  the 
RACER/ORCA  difficulty  factors.  From  [BTG  Incorporated  (Delta  Research 
Division).  Remedial  Action  Cost  Engineering  and  Requirements  (RACER)  System 
3.2.a.  Computer  Software  1998] 
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7.  Type  of  Operation 

The  “type  of  operation”  refers  to  the  survey  and  removal  methods  used  at  the  site. 
Traditional  removals  use  mag-and-flag  techniques.  More  modem  methods  use  global 
positioning  surveying  and  automated  recording  and  mapping.  The  OED  needs  to  track 
the  type  of  removal  as  a  predictor  of  costs.  Although  beyond  the  scope  of  this  thesis,  this 
data  will  also  be  important  in  the  future  for  justifying  investment  strategies.  The  OED 
should  use  the  RACER/ORCA  categories;  Table  20  shows  these  fields.  The  OED  may 
need  to  increase  the  categories  if  some  of  the  proposed  technologies  prove  useable 
(ground  penetrating  radar,  synthetic  dog  noses,  acoustic,  etc.). 


TYPE_OPERATION 

MEANING 

Conventional 

Mag-and-flag 

HHGPS 

Hand  held  GPS  used.  Manual  Recording 

GPS  Auto 

GPS  surveying  with  automated  removal  and 
mapping 

Table  20:  Recommended  Type  of  Operation  Categories  and  Their  Meanings.  The 
OED  should  track  the  type  of  cleanup  operation  performed  on  an  OOU.  The  table 
lists  the  recommended  inputs  for  this  Held.  From  [BTG  Incorporated  (Delta 
Research  Division).  Remedial  Action  Cost  Eneineerine  and  Requirements 
(RACER)  System  3.2.a.  Computer  Software  1998] 

B.  COST  INFORMATION 

Huntsville  Center  requested  that  the  OED  have  a  72-item  work  breakdown 
stmcture  for  reporting  cost  information.  The  contractor  responsible  for  populating  the 
OED  stated  that  finding  cost  information  in  this  detail  was  umealistic  since  no 
requirement  exists  for  contractors  to  report  cost  information  in  this  detail  [Chandler 
1999].  A  more  detailed  cost  structure  is  desirable,  but  this  thesis  requires  the  following 
fields  at  a  minimum. 
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1.  Date  Fields  for  Cleanup  Phases 

The  database  needs  to  add  date  fields  so  that  cost  information  can  be  normalized 
for  inflation.  The  database  needs  to  use  year  that  the  phase  (Preliminary  Assessment, 
Archive  Search  Report,  Engineering  Evaluation/Cost  Analysis,  and  Removal  Action)  was 
completed. 

2.  Preliminary  Assessment  and  Archive  Search  Report  Total  Cost 

Preliminary  Assessments  and  Archive  Search  Reports  usually  have  fixed  prices. 
These  costs  usually  represent  less  than  20%  of  the  cost  to  complete  and  therefore  require 
a  single  total  cost  entry. 

3.  Labor,  Material,  and  Travel/per  Diem  Costs  for  EE/CA  and  RA 

The  majority  of  the  cost  to  complete  occurs  in  the  EE/CA  and  RA  phases.  The 
OED,  at  a  minimum,  needs  to  report  costs  for  these  two  phases  by  “labor,”  “material,” 
and  “travel/per  Diem”  costs. 
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APPENDIX  B.  SCATTER  PLOTS  OF  SECONDARY  INDICATORS  OF  COST 


This  appendix  contains  the  scatter  plots  of  the  OOU  characteristics  that  the  thesis 
considers  as  secondary  indicators  of  cost.  These  plots  do  not  show  any  discemable 
relationship  between  the  OOU  descriptive  characteristics  and  cost  of  ordnance  removal. 
The  “secondary  indicators”  are  topography,  ordnance  type,  and  soil  type.  Figure  40 
shows  the  plots  of  the  “secondary  indicators.” 
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Figure  40:  Scatter  Plots  of  Secondary  Cost  Indicators.  The  secondary  cost 
indicators  include  soil,  topography,  and  ordnance  difficulty  factors.  These  cost 
indicators  do  not  show  expected  cost  correlations.  The  causes  of  these  unexpected 
results  may  be  due  to  interactions  with  and  overshadowing  by  the  primary 
indicators. 
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APPENDIX  C.  SCENARIO  OUTPUT  REPORT 
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Table  21;  Output  Report  For  the  Scenario  Cleanup.  The  report  shows  the 
summary  statistics  and  selection  criteria  for  the  scenario  cleanup. 
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APPENDIX  D.  SIMULATION  SUMMARY  FOR  THE  THREE-OOU  CLEANUP 


Statistic 

Value 

Minimum  = 

$716,691 

Maximum  = 

$2,412,688 

Mssamm 

$1,712,636 

Std  Deviation  = 

$304,859 

Mode  = 

$1,776,624 

Percentile 

Value 

5%  Perc  = 

$1,192,057 

10%  Perc  = 

$1,303,245 

15%  Perc  = 

$1,384,165 

20%  Perc  = 

$1,452,002 

25%  Perc  = 

$1,507,119 

30%  Perc  = 

$1,550,193 

35%  Perc  = 

$1,587,742 

40%  Perc  = 

$1,630,812 

45%  Perc  = 

$1,671,128 

50%  Perc  = 

$1,727,992 

55%  Perc  = 

$1,775,038 

60%  Perc  = 

$1,803,516 

65%  Perc  = 

$1,841,867 

70%  Perc  = 

$1,886,668 

75%  Perc  = 

$1,942,782 

80%  Perc  = 

$1,997,554 

85%  Perc  = 

$2,048,605 

90%  Perc  = 

$2,102,828 

95%  Perc  = 

$2,187,061 

Table  22:  Simulation  Summary  for  the  Three-OOU  Cleanup.  This  table  shows  the 
summary  statistics  and  the  cumulative  density  function  for  the  cost  of  the  three- 
OOU  cleanup.  The  distribution  is  nearly  symmetrical  with  the  mean,  median,  and 
mode  all  roughly  $1.7  million. 
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APPENDIX  E.  72-ITEM  WORK  BREAKDOWN  STRUCTURE 


Work  Breakdown 

Description 

Structure  ID 

32 


321 

322 


3221 

322102 


322103 

322105 


322111 


322112 


322113 


322114 


322115 


322116 

3222 


3223 

322302 


322303 


322304 


322305 


322311 


322312 

322313 


322314 


322315 

322316 


3224 


322403 


322404 

322406 


322411 

322412 


322413 

322414 


322415 

322416 


322417 

“1225 


322502 


322503 

322504 


322505 

322511 


322512 


322513 


322514 

322515 


Document 

Program  Management 


Project  Management 
EE/CA 


In-House  Work- Agreements 
Project  Support  Product  Services 


Woric  Plans 
Field  Investigations 


Formulation  and  Evaluation  of  RA  Alternatives 


Draft  Investigation  Report 


Final  Investigation  Report 


Value  Engineering  Studies 


Decision  Documents  and  Related  Products 


Other 

AE  and  Professional  Services  Procurement 


AE  and  Professional  Services  Contracts 
Project  Support  Products/Services 


Work  Plans 


Records  Research 


Field  Investigations 


Formulation  and  Evaluation  of  R  A  Alternatives 


Draft  Investigation  Report 
Final  Investigation  Report 


Value  Engineering  Studies 


Decision  Documents  and  Related  Products 
Other 


AE  Supervision  and  Review 
Project  Support  Products 


Work  Plans 
Field  Investigations 


Final  Risk  Assessment  Report 
Formulation  and  Evaluation  of  RA  Alternatives 


Draft  Investigation  Report 
Final  Investigation  Report 


Value  Engineering  Studies 

Decision  Documents  and  Related  Products 


Other 

Customer  Technical  Assistance 


Project  Support  Products/Services 


Work  Plans 
Records  Research 


Field  Investigations 

Formulation  and  Evaluation  of  RA  Alternatives 


Draft  Investigation  Report 


Final  Investigation  Report 


Value  Engineering  Studies 

Decision  Documents  and  Related  Products 


322516 

Other 

33 

Remediation  Action 

3301 

Mobilization  and  Preparatory  Work 

3302 

Monitoring,  Sampling,  Testing,  and  Analysis 

3303 

Site  Work 

3304 

OE  Removal  and  Destruction 

330401 

Community  Relations  Meetings 

330402 

Location  Surveys  and  Mapping 

330403 

Ordnance  Removal  and  Destruction 

3318 

OE  Disposal 

3320 

Site  Restoration 

3321 

Demobilization 

3322 

General  Requirements 

332201 

Supervision  and  Management 

332202 

Administrative  Job  Office 

332203 

Warehouse,  Materials  Handling,  and  Purchasing 

332204 

‘  Engineering,  Surveying,  and  Quality  Control 

332205 

Equipment  Maintenance  and  Motor  Pool 

332206 

First  Aid,  Fire  Protection,  Traffic  Control,  and  Security 

332207 

Health  and  Safety 

332208 

Temporary  Construction  Facilities  ~  Ownership 

332209 

Temporary  Construction  Facilities  -  Operation 

332210 

Project  Utilities 

332211 

Misc.  Project  Expenses 

332212 

Insurance,  Interest,  and  Fees 

Table  23:  The  72-Item  Work  Breakdown  Structure.  The  OED  cost  reporting 
structure  incorporates  this  work  breakdown  structure.  The  structure  is 
hierarchical  and  indents  subtasks  of  primary  tasks.  Populating  the  OED  with  cost 
data  proves  difficult  because  contractors  rarely  report  data  with  this  fidelity.  This 
thesis  uses  a  more  aggregated  cost  breakdown  based  on  available  cost  data. 


104 


LIST  OF  REFERENCES 


Aeronautical  Systems  Center.  Funding  Estimator  Tool  for  Weapon  System  Pollution 
Prevention.  Draft  Technical  Report  prepared  for  Air  Force  Material  Command 
Acquisition  Pollution  Prevention  Tools  Integrated  Product  Team.  July  1994. 

AnalyCorp.  Insight.xla.  Computer  Software,  1998. 

Anderson,  T.  and  J.  Cherwonik.  “Cost  Estimating  Risk  and  Cost  Estimating  Uncertainity 
Guidelines.”  Acquisition  Review  Quarterly  (1997):  339  -  347. 

Barrett,  B.  “Calculation  of  Minimum  Detectable  Statistical  Level  of  the  Current  Grid 
Stats/Site  Stats  Method.”  Final  Report  prepared  for  United  States  Army  Engineering  and 
Support  Center  Huntsville,  Alabama.  January  1999. 

Book,  S.  “The  Probabilistic  Approach  to  Cost  Analysis.”  Guest  lecture  presented  to  the 
Naval  Postgraduate  School.  Monterey,  California.  March  1999. 

BTG  Incorporated  (Delta  Research  Division). 

[http://www.deltabtg.com/enviro/racer.main.html].  accessed  Febmary  1999. 

BTG  Incorporated  (Delta  Research  Division).  Remedial  Action  Cost  Engineering  and 
Requirements  (RACER)  System  3.2.a.  Computer  Software,  1998. 

Chandler,  K.  (OED  Contractor,  Computer  Systems  Technology),  phone  conversation 
March  1999. 

Computer  Systems  Technology.  Ordnance  and  Explosives  Data  Base.  Computer 
Software,  1996. 

Computer  Systems  Technology.  Ordnance  and  Explosives  Data  Base.  Draft  Final 
Report  prepared  for  United  States  Army  Engineering  and  Support  Center  Huntsville, 
Alabama,  March  1996. 

Conover,  W.  Practical  Nonparametric  Statistics.  New  York.  John  Wiley  and  Sons  Inc., 
1999. 

Decisioneering  Inc.  Crystal  Ball.  Computer  Software,  1997. 

Fanning,  A.  (Analyst  United  States  Army  Engineering  and  Support  Center),  personal 
interview.  March  1999. 


105 


Foley,  J.,  Gifford,  M.,  Millhouse,  S.,  and  L.  Helms.  "Ordnance  and  Explosives  Program, 
Geographic  Information  System  and  Knowledge  Base.” 
[http://www.scainc.com/oegiskb.htm].  accessed  December  1998. 

Hersey,  J.  and  A.  Pederson.  "Range  Cost  Model."  Unpublished  report  for  the  Navy 
Explosives  Ordnance  Disposal  Technology  Division,  1981. 

Hughitt,  E.,  Wusterbarth  A.,  and  M.  Zoepfl.  “An  Economic  Analysis  Methodology  for 
Comparing  Alternative  Chemical  Demilitarization  Processes.”  28*  Annual  DOD  Cost 
Analysis  Symposium.  Septemper  1994. 

Lurie,  P.  and  M.  Goldberg.  “An  Approximate  Method  for  Sampling  Correlated  Random 
Variables  from  partially-specified  Distributions.”  Management  Science.  44(1998):  203- 
218. 

Meuser,  M,  and  A.  Szasz.  "Stakeholder  Participation  in  the  Toxic  Cleanup  of  Military 
Facilities  and  its  Relationship  to  the  Prospects  for  Economic  Reuse:  The  Case  of  Fort 
Ord,  CA."  Journal  of  Contemporary  Sociology.  34  (1997):  212-222. 

Microsoft  Corporation.  Excel  97.  Computer  Software,  1997. 

Microsoft  Corporation.  Visual  Basic.  Computer  Software,  1997. 

Palisades  Corporation.  ©Risk.  Computer  Software,  1997. 

Panama  Canal  Treaty  Implementation  Plan  Agency.  Evaluation  of  Unexploded  Ordnance 
Detection  and  Interrogation  Technologies.  For  use  in  Panama:  Empire.  Balboa  West,  and 
Pina  Ranges.  Washington,  D.C.,  1997. 

Rohrer,  W.  Adak  Ordnance  Management  Plan:  The  Case  for  Institutional  Controls  and 
Alternative  UXO  Clearance  Depths.  Seattle,  Washington:  GRS  Greiner  Inc.,  1997. 

United  States  Department  of  Defense  ,  Office  of  the  Secretary.  Closed.  Transferred,  and 
Transferring  Ranges  Containing  Military  Munitions:  Proposed  Rule.  32  CFR  Part  178. 
Washington  D.C.,  1997. 

United  States  Department  of  Defense,  Office  of  the  Secretary.  DOD  Ammunition  and 
Explosive  Safety  Standards.  DOD  6055.9-STD.  Washington  D.C.,  1995. 

United  States  Department  of  Defense,  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology.  Report  of  the  Defense  Science  Board  Task  Force  on 
Unexploded  Ordnance  (UXO)  Clearance.  Active  Range  UXO  Clearance  and  Explosive 
Ordnance  Disposal  (EOD)  Programs.  Washington  D.C.,  1998. 


106 


United  States  Department  of  Defense,  Office  of  Under  Secretary  of  Defense  for 
Acquisition  and  Technology.  Report  to  Congress:  Unexploded  Ordnance  Clearance:  A 
rnnrdinated  Approach  to  Requirements  and  Technology  Development.  Washington 
D.C.,  1997. 


Young,  M.  (Cost  Analyst  United  States  Army  Engineering  and  Support  Center, 
Huntsville),  personal  interview.  March  1999. 


107 


108 


INITIAL  DISTRIBUTION  LIST 


1 .  Defense  T echnical  Information  Center . 2 

8725  John  J.  Kingman  Rd.,  STE  0944 

Fort  Belvoir,  Virginia  22060-6218 

2.  Dudley  Knox  Library . 2 

Naval  Postgraduate  School 

411  Dyer  Rd. 

Monterey,  California  93943-5 101 

3.  Director,  Training  and  Education . 1 

MCCDC,  Code  C46 

1019  Elliot  Rd. 

Quantico,  Virginia  22134-5027 

4.  Director,  Marine  Corps  Research  Center . 1 

MCCDC,  Code  C40RC 

2040  Broadway  Street 
Quantico,  Virginia  22134-5107 

5.  Director,  Studies  and  Analysis  Division . 1 

MCCDC,  Code  C45 

300  Russell  Road 
Quantico,  Virginia  22134-5130 

6.  Director,  Naval  Center  for  Cost  Analysis . 2 

3801  Nebraska  Avenue,  N.W. 

Washington,  DC  20393-5444 

7.  LCDR  Timothy  P.  Anderson,  Code  OR/at . 2 

Department  of  Operations  Research 

Naval  Postgraduate  School 
Monterey,  California  93943-5000 

8.  Professor  Robert  F.  Dell,  Code  OR/de . 2 

Department  of  Operations  Research 

Naval  Postgraduate  School 
Monterey,  California  93943-5000 


109 


9.  Mrs.  Kelley  Dreyer . 2 

Headquarters  Marine  Corps  (LFL) 

2  Navy  Annex,  Room  3110 
Washington,  DC  20380 

10.  Mr.  Arkie  Fanning . 2 

Department  of  the  Army 

United  States  Army  Engineering  and  Support  Center 
P.O.  Box  1600 

Huntsville,  Alabama  35807-4301 

1 1 .  Maj  Peter  L.  Poppe . 2 

3005  Clippership  Dr. 

Stafford,  Virginia  22554 


no 


